[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:31 dist-test-slave-nq8r krb5kdc[19730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:31 dist-test-slave-nq8r krb5kdc[19730](info): set up 2 sockets
Jun 20 19:12:31 dist-test-slave-nq8r krb5kdc[19730](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 19:12:32.761510 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:32 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:32.865353 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33401
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:33401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:32.942731 19743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:32.942929 19743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:32.942948 19743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:32.944059 19743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:32.944099 19743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:32.944113 19743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:32.944136 19743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:32.944159 19743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:32.945636 19743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:33401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:32.945845 19743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:32.946069 19743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:32.950810 19751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:32.950855 19749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:32.950873 19748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:32.950965 19743 server_base.cc:1034] running on GCE node
I20260620 19:12:32.951251 19743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:32.951524 19743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:32.952661 19743 hybrid_clock.cc:648] HybridClock initialized: now 1781982752952650 us; error 26 us; skew 500 ppm
Jun 20 19:12:32 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982752, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:32.954937 19743 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:32.955355 19743 webserver.cc:460] Webserver started at http://127.19.66.254:34609/ using document root <none> and password file <none>
I20260620 19:12:32.955546 19743 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:32.955590 19743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:32.955724 19743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:32.956732 19743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "caacadcc60c84f308030ec0f31e36ab6"
format_stamp: "Formatted at 2026-06-20 19:12:32 on dist-test-slave-nq8r"
server_key: "852ab0cdbb79a48edfbfcdfe1d92ad57"
server_key_iv: "915eeaf9f28cfb0887e15f7d150eb68b"
server_key_version: "encryptionkey@0"
I20260620 19:12:32.957095 19743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "caacadcc60c84f308030ec0f31e36ab6"
format_stamp: "Formatted at 2026-06-20 19:12:32 on dist-test-slave-nq8r"
server_key: "852ab0cdbb79a48edfbfcdfe1d92ad57"
server_key_iv: "915eeaf9f28cfb0887e15f7d150eb68b"
server_key_version: "encryptionkey@0"
I20260620 19:12:32.958359 19743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:32.959003 19758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:32.959132 19743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:32.959209 19743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "caacadcc60c84f308030ec0f31e36ab6"
format_stamp: "Formatted at 2026-06-20 19:12:32 on dist-test-slave-nq8r"
server_key: "852ab0cdbb79a48edfbfcdfe1d92ad57"
server_key_iv: "915eeaf9f28cfb0887e15f7d150eb68b"
server_key_version: "encryptionkey@0"
I20260620 19:12:32.959262 19743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:32.966010 19743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:32.967569 19743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:32.967667 19743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:32.971300 19743 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:33401
I20260620 19:12:32.971331 19810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:33401 every 8 connection(s)
I20260620 19:12:32.971625 19743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:32.972409 19811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:32.975167 19811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: Bootstrap starting.
I20260620 19:12:32.975860 19811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:32.976162 19811 log.cc:824] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:32.976933 19811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: No bootstrap required, opened a new log
I20260620 19:12:32.979041 19811 raft_consensus.cc:348] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } }
I20260620 19:12:32.979238 19811 raft_consensus.cc:374] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:32.979281 19811 raft_consensus.cc:729] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caacadcc60c84f308030ec0f31e36ab6, State: Initialized, Role: FOLLOWER
I20260620 19:12:32.979410 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 19743
I20260620 19:12:32.979430 19811 consensus_queue.cc:260] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } }
I20260620 19:12:32.979509 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:32.979511 19811 raft_consensus.cc:388] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:32.979543 19811 raft_consensus.cc:482] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:32.979583 19811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:32.980248 19723 external_mini_cluster.cc:1428] Setting key af009ae791538ea4f595e7d437b8877d
I20260620 19:12:32.980295 19811 raft_consensus.cc:504] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } }
I20260620 19:12:32.980401 19811 leader_election.cc:302] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caacadcc60c84f308030ec0f31e36ab6; no voters:
I20260620 19:12:32.980546 19811 leader_election.cc:288] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:32.980581 19814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:32.980690 19814 raft_consensus.cc:686] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [term 1 LEADER]: Becoming Leader. State: Replica: caacadcc60c84f308030ec0f31e36ab6, State: Running, Role: LEADER
I20260620 19:12:32.980782 19811 sys_catalog.cc:564] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:32.980765 19814 consensus_queue.cc:237] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } }
I20260620 19:12:32.981438 19814 sys_catalog.cc:455] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caacadcc60c84f308030ec0f31e36ab6. Latest consensus state: current_term: 1 leader_uuid: "caacadcc60c84f308030ec0f31e36ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } } }
I20260620 19:12:32.981515 19814 sys_catalog.cc:458] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:32.982591 19815 sys_catalog.cc:455] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caacadcc60c84f308030ec0f31e36ab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caacadcc60c84f308030ec0f31e36ab6" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33401 } } }
I20260620 19:12:32.982676 19815 sys_catalog.cc:458] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6 [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:32.982811 19831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:32.982856 19831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:32.983381 19824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:32.983544 19824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:32.984948 19824 catalog_manager.cc:1347] Generated new cluster ID: bfa4409b43d34a49b02a6509a0f76052
I20260620 19:12:32.984995 19824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:32 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:32.997681 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:32.981104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.19.66.254:33401)
0620 19:12:32.981318 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:32.981329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:32.981606 (+ 277us) server_negotiation.cc:407] Connection header received
0620 19:12:32.983836 (+ 2230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:32.983843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:32.983909 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:32.984043 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:32.986051 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:32.987421 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:32.988969 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:32.989571 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:32.989665 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:32.993834 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:32.993838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:32.993839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:32.993850 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:32.995760 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:32.996209 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:32.996212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:32.996212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:32.996328 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:32.996558 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:32.996559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:32.996559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:32.996897 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:32.997027 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:32.997307 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 19:12:32.997444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":58,"threads_started":1}
I20260620 19:12:33.009860 19824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:33.010356 19824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:33.015184 19824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caacadcc60c84f308030ec0f31e36ab6: Generated new TSK 0
I20260620 19:12:33.015340 19824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:33.038687 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:33.139022 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:33.139214 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:33.139235 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:33.140303 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:33.140336 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:33.140362 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:33.141776 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:33.141952 19837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:33.142119 19837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:33.146595 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:33.146698 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.146737 19837 server_base.cc:1034] running on GCE node
W20260620 19:12:33.146593 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.146955 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:33.147190 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:33.148325 19837 hybrid_clock.cc:648] HybridClock initialized: now 1781982753148308 us; error 34 us; skew 500 ppm
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:33.150681 19837 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:33.151113 19837 webserver.cc:460] Webserver started at http://127.19.66.193:44103/ using document root <none> and password file <none>
I20260620 19:12:33.151280 19837 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:33.151319 19837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:33.151415 19837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:33.152345 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "7a90943c4ee4490fbd5d7bdc2f0aaab2"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "c8447ad365ea14cb7133b0246557b205"
server_key_iv: "e53e807c059ff1489e834587dddc894e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.152632 19837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "7a90943c4ee4490fbd5d7bdc2f0aaab2"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "c8447ad365ea14cb7133b0246557b205"
server_key_iv: "e53e807c059ff1489e834587dddc894e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.153828 19837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:33.154421 19852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.154557 19837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:33.154633 19837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "7a90943c4ee4490fbd5d7bdc2f0aaab2"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "c8447ad365ea14cb7133b0246557b205"
server_key_iv: "e53e807c059ff1489e834587dddc894e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.154686 19837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:33.165532 19837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:33.167120 19837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:33.167222 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:33.167397 19837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:33.167629 19837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:33.167656 19837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.167676 19837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:33.167692 19837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.172729 19837 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:43261
I20260620 19:12:33.172798 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:43261 every 8 connection(s)
I20260620 19:12:33.173082 19837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:33.173223 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 19837
I20260620 19:12:33.173310 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:33.173456 19723 external_mini_cluster.cc:1428] Setting key e26e50f94fc03ee15b199a0e4f7d982f
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
I20260620 19:12:33.182525 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.173743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:43283 (local address 127.19.66.254:33401)
0620 19:12:33.173785 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.173787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.173987 (+ 200us) server_negotiation.cc:407] Connection header received
0620 19:12:33.174580 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.174580 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.174592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.174639 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.175761 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.176478 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.177497 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.177805 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.177878 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.179787 (+ 1909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.179789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.179789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.179797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.181306 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.181656 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.181657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.181657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.181703 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.181962 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.181962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.181963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.182058 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.182103 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.182374 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.182456 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:33.182797 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.173737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33401 (local address 127.19.66.193:43283)
0620 19:12:33.173886 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:33.173919 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:33.174481 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:33.174711 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:33.174715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:33.175418 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:33.175670 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.175686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.176555 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.176556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.177414 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.177418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.177887 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.177887 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.177987 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.178445 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:33.178452 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:33.179701 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:33.181369 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.181376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.181378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.181601 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.181812 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.181813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.181813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.181918 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.182101 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:33.182104 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:33.182245 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 19:12:33.182343 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:12:33.182531 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
I20260620 19:12:33.183384 19966 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33401
I20260620 19:12:33.183473 19966 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:33.183640 19966 heartbeater.cc:511] Master 127.19.66.254:33401 requested a full tablet report, sending...
I20260620 19:12:33.184142 19775 ts_manager.cc:194] Registered new tserver with Master: 7a90943c4ee4490fbd5d7bdc2f0aaab2 (127.19.66.193:43261)
I20260620 19:12:33.184687 19775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:43283
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:33.196435 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:33.272965 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:33.273154 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:33.273175 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:33.274250 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:33.274282 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:33.274307 19971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:33.275635 19971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:33.275835 19971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:33.276002 19971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:33.280514 19979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:33.280521 19976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.280651 19971 server_base.cc:1034] running on GCE node
W20260620 19:12:33.280529 19977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.281045 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:33.281270 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:33.282405 19971 hybrid_clock.cc:648] HybridClock initialized: now 1781982753282393 us; error 30 us; skew 500 ppm
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:33.284758 19971 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:33.285214 19971 webserver.cc:460] Webserver started at http://127.19.66.194:40569/ using document root <none> and password file <none>
I20260620 19:12:33.285405 19971 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:33.285451 19971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:33.285573 19971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:33.286568 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "5d4eabd9416f4758ad86a1abdf22ab00"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "b2a447b8fb9c5fdf69423a10026a0a09"
server_key_iv: "3b76aede9d6586ba02114e4ba42b1402"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.286904 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "5d4eabd9416f4758ad86a1abdf22ab00"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "b2a447b8fb9c5fdf69423a10026a0a09"
server_key_iv: "3b76aede9d6586ba02114e4ba42b1402"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.288154 19971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:12:33.288785 19986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.288949 19971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:33.289028 19971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "5d4eabd9416f4758ad86a1abdf22ab00"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "b2a447b8fb9c5fdf69423a10026a0a09"
server_key_iv: "3b76aede9d6586ba02114e4ba42b1402"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.289093 19971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:33.303442 19971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:33.305091 19971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:33.305219 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:33.305416 19971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:33.305692 19971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:33.305725 19971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.305752 19971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:33.305768 19971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.311094 19971 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:37497
I20260620 19:12:33.311133 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:37497 every 8 connection(s)
I20260620 19:12:33.311405 19971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:33.319566 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.312026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:34171 (local address 127.19.66.254:33401)
0620 19:12:33.312071 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.312073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.312258 (+ 185us) server_negotiation.cc:407] Connection header received
0620 19:12:33.312715 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.312716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.312724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.312778 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.313682 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.314177 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.315087 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.315266 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.315308 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.317193 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.317194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.317194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.317200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.318397 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.318738 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.318739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.318740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.318790 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.318992 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.318993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.318993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.319098 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.319153 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.319395 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.319476 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:33.319720 20102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.311981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33401 (local address 127.19.66.194:34171)
0620 19:12:33.312140 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:33.312179 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:33.312639 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:33.312864 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:33.312869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:33.313431 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:33.313614 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.313618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.314236 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.314237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.315026 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.315028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.315323 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.315324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.315414 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.315832 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:33.315840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:33.317101 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:33.318470 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.318476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.318478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.318674 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.318841 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.318841 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.318842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.318940 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.319149 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:33.319156 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:33.319282 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 19:12:33.319367 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:12:33.319517 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260620 19:12:33.320125 20100 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33401
I20260620 19:12:33.320214 20100 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:33.320372 20100 heartbeater.cc:511] Master 127.19.66.254:33401 requested a full tablet report, sending...
I20260620 19:12:33.320670 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 19971
I20260620 19:12:33.320711 19775 ts_manager.cc:194] Registered new tserver with Master: 5d4eabd9416f4758ad86a1abdf22ab00 (127.19.66.194:37497)
I20260620 19:12:33.320745 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:33.320926 19723 external_mini_cluster.cc:1428] Setting key 988e6d92d1b675f54368103a28402023
I20260620 19:12:33.321144 19775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:34171
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:33.341024 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:33.417491 20105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:33.417687 20105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:33.417709 20105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:33.418785 20105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:33.418819 20105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:33.418843 20105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:33.420217 20105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33401
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:33.420405 20105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:33.420583 20105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:33.425179 20110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:33.425192 20113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.425277 20105 server_base.cc:1034] running on GCE node
W20260620 19:12:33.425211 20111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:33.425532 20105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:33.425735 20105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:33.426867 20105 hybrid_clock.cc:648] HybridClock initialized: now 1781982753426852 us; error 29 us; skew 500 ppm
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:33.429299 20105 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:33.429708 20105 webserver.cc:460] Webserver started at http://127.19.66.195:34219/ using document root <none> and password file <none>
I20260620 19:12:33.429924 20105 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:33.429971 20105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:33.430096 20105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:33.431149 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "8676cf72e35b4fec8b4801f8e233a691"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "24a88f8b7c8b9aca466410077b947360"
server_key_iv: "e7b107b97cdfa62030b2b02cc6968a1e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.431479 20105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "8676cf72e35b4fec8b4801f8e233a691"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "24a88f8b7c8b9aca466410077b947360"
server_key_iv: "e7b107b97cdfa62030b2b02cc6968a1e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.432641 20105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:33.433323 20120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.433465 20105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.433540 20105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "8676cf72e35b4fec8b4801f8e233a691"
format_stamp: "Formatted at 2026-06-20 19:12:33 on dist-test-slave-nq8r"
server_key: "24a88f8b7c8b9aca466410077b947360"
server_key_iv: "e7b107b97cdfa62030b2b02cc6968a1e"
server_key_version: "encryptionkey@0"
I20260620 19:12:33.433604 20105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:33.450248 20105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:33.452411 20105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:33.452543 20105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:33.452752 20105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:33.453080 20105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:33.453114 20105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.453143 20105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:33.453166 20105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:33.458607 20105 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:37469
I20260620 19:12:33.458669 20233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:37469 every 8 connection(s)
I20260620 19:12:33.458935 20105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:33.464969 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20105
I20260620 19:12:33.465066 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:33.465241 19723 external_mini_cluster.cc:1428] Setting key 0e82a5a156a1b0e06c4e3a2d51be594a
I20260620 19:12:33.467339 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.459568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:36111 (local address 127.19.66.254:33401)
0620 19:12:33.459612 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.459615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.459841 (+ 226us) server_negotiation.cc:407] Connection header received
0620 19:12:33.460332 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.460333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.460343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.460394 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.461346 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.461843 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.462877 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.463058 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.463106 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.465107 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.465108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.465109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.465117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.466385 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.466708 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.466709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.466709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.466753 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.466905 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.466906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.466906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.466994 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.467038 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.467226 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.467277 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:33.467535 20236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.459547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33401 (local address 127.19.66.195:36111)
0620 19:12:33.459711 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:33.459750 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:33.460221 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:33.460450 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:33.460453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:33.461045 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:33.461262 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.461268 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.461910 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.461911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.462814 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:33.462816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.463114 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.463115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:33.463213 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.463687 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:33.463694 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:33.465006 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:33.466447 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.466452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.466453 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.466653 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.466786 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:33.466786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:33.466787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:33.466867 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:33.467035 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:33.467037 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:33.467141 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:12:33.467215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:12:33.467339 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260620 19:12:33.467928 20234 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33401
I20260620 19:12:33.468014 20234 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:33.468148 20234 heartbeater.cc:511] Master 127.19.66.254:33401 requested a full tablet report, sending...
I20260620 19:12:33.468472 19774 ts_manager.cc:194] Registered new tserver with Master: 8676cf72e35b4fec8b4801f8e233a691 (127.19.66.195:37469)
I20260620 19:12:33.468869 19774 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:36111
I20260620 19:12:33.476981 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:33.498448 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.491225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.19.66.193:43261)
0620 19:12:33.491394 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.491398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.491406 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:33.491429 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.491430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.491443 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.491505 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.492086 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.492810 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.493525 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.493845 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.493903 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.495910 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.495919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.495919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.495927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.497203 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.497501 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.497504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.497504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.497554 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.497932 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.497933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.497934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.498061 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.498189 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.498261 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.498334 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
W20260620 19:12:33.498761 19880 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53120
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[19730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:33.518674 20241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:33.512719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53128 (local address 127.19.66.193:43261)
0620 19:12:33.512784 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:33.512786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:33.512877 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:12:33.512963 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:33.512964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:33.512976 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:33.513023 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:33.513499 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.513967 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.514622 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:33.514801 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:33.514841 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:33.516781 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:33.516783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:33.516783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:33.516789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:33.517900 (+ 1111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.518171 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.518172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.518172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.518212 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:33.518332 (+ 120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:33.518332 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:33.518332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:33.518418 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:33.518496 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:33.518553 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:12:33.518606 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:12:33.519150 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 19837
I20260620 19:12:33.523201 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 19971
I20260620 19:12:33.527472 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20105
I20260620 19:12:33.530846 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 19743
2026-06-20T19:12:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1721 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[20248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[20248](info): set up 2 sockets
Jun 20 19:12:33 dist-test-slave-nq8r krb5kdc[20248](info): commencing operation
krb5kdc: starting...
W20260620 19:12:34.476104 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:34.580160 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46815
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:34.656419 20261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:34.656612 20261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:34.656641 20261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:34.657783 20261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:34.657823 20261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:34.657841 20261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:34.657857 20261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:34.657871 20261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:34.659301 20261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:34.659536 20261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:34.659729 20261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:34.663975 20269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.664034 20267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.664096 20266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:34.664346 20261 server_base.cc:1034] running on GCE node
I20260620 19:12:34.664577 20261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:34.664832 20261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:34.665988 20261 hybrid_clock.cc:648] HybridClock initialized: now 1781982754665974 us; error 32 us; skew 500 ppm
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:34.668223 20261 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:34.668649 20261 webserver.cc:460] Webserver started at http://127.19.66.254:41219/ using document root <none> and password file <none>
I20260620 19:12:34.668877 20261 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:34.668928 20261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:34.669054 20261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:34.670112 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "9a5828de11574d70903e8254e2cb492a"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "df99116e6acee1541972267601b30f4c"
server_key_iv: "453b5b6f30f50718223a3a181d2680f2"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.670428 20261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "9a5828de11574d70903e8254e2cb492a"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "df99116e6acee1541972267601b30f4c"
server_key_iv: "453b5b6f30f50718223a3a181d2680f2"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.671613 20261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:12:34.672271 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.672430 20261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.672493 20261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "9a5828de11574d70903e8254e2cb492a"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "df99116e6acee1541972267601b30f4c"
server_key_iv: "453b5b6f30f50718223a3a181d2680f2"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.672559 20261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:34.689385 20261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:34.690976 20261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:34.691094 20261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:34.694662 20261 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:46815
I20260620 19:12:34.694809 20328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:46815 every 8 connection(s)
I20260620 19:12:34.695005 20261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:34.695854 20329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:34.697988 20329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a: Bootstrap starting.
I20260620 19:12:34.698496 20329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:34.698717 20329 log.cc:824] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:34.699293 20329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a: No bootstrap required, opened a new log
I20260620 19:12:34.700819 20329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } }
I20260620 19:12:34.701015 20329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:34.701047 20329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5828de11574d70903e8254e2cb492a, State: Initialized, Role: FOLLOWER
I20260620 19:12:34.701157 20329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } }
I20260620 19:12:34.701206 20329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:34.701225 20329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:34.701256 20329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:34.701715 20329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } }
I20260620 19:12:34.701797 20329 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5828de11574d70903e8254e2cb492a; no voters:
I20260620 19:12:34.701913 20329 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:34.701956 20332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:34.702114 20332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [term 1 LEADER]: Becoming Leader. State: Replica: 9a5828de11574d70903e8254e2cb492a, State: Running, Role: LEADER
I20260620 19:12:34.702134 20329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:34.702255 20332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } }
I20260620 19:12:34.702632 20334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5828de11574d70903e8254e2cb492a. Latest consensus state: current_term: 1 leader_uuid: "9a5828de11574d70903e8254e2cb492a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } } }
I20260620 19:12:34.702713 20334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:34.702737 20333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5828de11574d70903e8254e2cb492a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5828de11574d70903e8254e2cb492a" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46815 } } }
I20260620 19:12:34.702806 20333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:34.703236 20341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:34.703719 20341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:34.703958 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20261
I20260620 19:12:34.704035 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:34.704155 19723 external_mini_cluster.cc:1428] Setting key f5b33b4440e4cb7e33580c5c2b992566
I20260620 19:12:34.705175 20341 catalog_manager.cc:1347] Generated new cluster ID: 9eaf8c19fb9a42e7b599088efefbe10f
I20260620 19:12:34.705264 20341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:34.714876 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:34.704518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55314 (local address 127.19.66.254:46815)
0620 19:12:34.704765 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:34.704773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:34.704806 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:34.704939 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:34.704946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:34.704999 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:34.705207 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:34.705864 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.707249 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.708228 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.708876 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.708983 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:34.711516 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:34.711519 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:34.711520 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:34.711531 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:34.713355 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:34.713754 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:34.713756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:34.713756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:34.713849 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:34.714027 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:34.714028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:34.714028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:34.714288 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:34.714417 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:34.714532 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:12:34.714656 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":209,"thread_start_us":67,"threads_started":1}
I20260620 19:12:34.722604 20341 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:34.723070 20341 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:34.724661 20341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5828de11574d70903e8254e2cb492a: Generated new TSK 0
I20260620 19:12:34.724828 20341 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:34.748363 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:34.826344 20355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:34.826536 20355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:34.826571 20355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:34.827698 20355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:34.827739 20355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:34.827773 20355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:34.829288 20355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:34.829512 20355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:34.829703 20355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:34.834177 20363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.834218 20361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.834177 20360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:34.834334 20355 server_base.cc:1034] running on GCE node
I20260620 19:12:34.834595 20355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:34.834820 20355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:34.835949 20355 hybrid_clock.cc:648] HybridClock initialized: now 1781982754835935 us; error 31 us; skew 500 ppm
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:34.838248 20355 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:34.838627 20355 webserver.cc:460] Webserver started at http://127.19.66.193:33871/ using document root <none> and password file <none>
I20260620 19:12:34.838824 20355 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:34.838864 20355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:34.838979 20355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:34.839919 20355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "adc5b3d524b5475db6f6c086b2839218"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "c75f4c48c0d22116bcb24573a60d78d3"
server_key_iv: "e022de4cf7c80eb981aee6a7cc5851da"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.840217 20355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "adc5b3d524b5475db6f6c086b2839218"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "c75f4c48c0d22116bcb24573a60d78d3"
server_key_iv: "e022de4cf7c80eb981aee6a7cc5851da"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.841471 20355 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:12:34.842100 20370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.842258 20355 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.842335 20355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "adc5b3d524b5475db6f6c086b2839218"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "c75f4c48c0d22116bcb24573a60d78d3"
server_key_iv: "e022de4cf7c80eb981aee6a7cc5851da"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.842397 20355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:34.848237 20355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:34.849902 20355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:34.850020 20355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:34.850196 20355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:34.850466 20355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:34.850497 20355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.850523 20355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:34.850548 20355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.855679 20355 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:32971
I20260620 19:12:34.855744 20483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:32971 every 8 connection(s)
I20260620 19:12:34.856021 20355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:34.862814 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20355
I20260620 19:12:34.862908 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:34.863080 19723 external_mini_cluster.cc:1428] Setting key ed756662eaf80b3c96986f598c2752f9
I20260620 19:12:34.864732 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:34.856644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:55089 (local address 127.19.66.254:46815)
0620 19:12:34.856697 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:34.856699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:34.856887 (+ 188us) server_negotiation.cc:407] Connection header received
0620 19:12:34.857786 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:34.857787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:34.857799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:34.857851 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:34.858444 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.858951 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.859913 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.860123 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.860178 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:34.862146 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:34.862147 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:34.862148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:34.862155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:34.863460 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:34.863881 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:34.863882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:34.863882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:34.863939 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:34.864179 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:34.864179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:34.864179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:34.864276 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:34.864319 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:34.864597 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 19:12:34.864669 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:12:34.865029 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:34.856618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46815 (local address 127.19.66.193:55089)
0620 19:12:34.856745 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:34.856783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:34.857683 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:34.857922 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:34.857926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:34.858168 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:34.858353 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:34.858357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.859028 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.859030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:34.859821 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:34.859823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.860191 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.860192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:34.860308 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:34.860719 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:34.860727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:34.862062 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:34.863524 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:34.863531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:34.863533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:34.863804 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:34.863995 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:34.863995 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:34.863996 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:34.864125 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:34.864321 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:34.864325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:34.864472 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 19:12:34.864570 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:12:34.864720 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260620 19:12:34.865433 20484 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46815
I20260620 19:12:34.865518 20484 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:34.865684 20484 heartbeater.cc:511] Master 127.19.66.254:46815 requested a full tablet report, sending...
I20260620 19:12:34.866133 20293 ts_manager.cc:194] Registered new tserver with Master: adc5b3d524b5475db6f6c086b2839218 (127.19.66.193:32971)
I20260620 19:12:34.866719 20293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:55089
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:34.882879 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:34.958957 20489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:34.959151 20489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:34.959179 20489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:34.960274 20489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:34.960312 20489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:34.960347 20489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:34.961791 20489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:34.962021 20489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:34.962214 20489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:34.966861 20494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.966871 20497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:34.967016 20495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:34.967043 20489 server_base.cc:1034] running on GCE node
I20260620 19:12:34.967228 20489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:34.967444 20489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:34.968575 20489 hybrid_clock.cc:648] HybridClock initialized: now 1781982754968559 us; error 27 us; skew 500 ppm
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:34.970950 20489 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:34.971367 20489 webserver.cc:460] Webserver started at http://127.19.66.194:34593/ using document root <none> and password file <none>
I20260620 19:12:34.971546 20489 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:34.971585 20489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:34.971673 20489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:34.972628 20489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "7ce634ace0c2491897a3eaa9f685c13e"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "5c57b278cf1ef73bc320eb6b1db9cf76"
server_key_iv: "2b9ad0d0d89f48e521b39158ce9cfb04"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.972965 20489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "7ce634ace0c2491897a3eaa9f685c13e"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "5c57b278cf1ef73bc320eb6b1db9cf76"
server_key_iv: "2b9ad0d0d89f48e521b39158ce9cfb04"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.974215 20489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:34.975052 20504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.975224 20489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:34.975304 20489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "7ce634ace0c2491897a3eaa9f685c13e"
format_stamp: "Formatted at 2026-06-20 19:12:34 on dist-test-slave-nq8r"
server_key: "5c57b278cf1ef73bc320eb6b1db9cf76"
server_key_iv: "2b9ad0d0d89f48e521b39158ce9cfb04"
server_key_version: "encryptionkey@0"
I20260620 19:12:34.975368 20489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:34.984678 20489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:34.986694 20489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:34.986801 20489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:34.987020 20489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:34.987303 20489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:34.987334 20489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.987362 20489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:34.987386 20489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:34.992654 20489 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:41053
I20260620 19:12:34.992702 20617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:41053 every 8 connection(s)
I20260620 19:12:34.993017 20489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:34.996816 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20489
I20260620 19:12:34.996912 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:34.997067 19723 external_mini_cluster.cc:1428] Setting key 767d9852e534dd11e90ac1413793e55c
Jun 20 19:12:34 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:35.001935 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:34.993625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50351 (local address 127.19.66.254:46815)
0620 19:12:34.993700 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:34.993703 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:34.993865 (+ 162us) server_negotiation.cc:407] Connection header received
0620 19:12:34.994719 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:34.994720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:34.994729 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:34.994777 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:34.995324 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.995839 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.996825 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.997090 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.997146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:34.999406 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:34.999408 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:34.999408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:34.999415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.000625 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.001039 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.001040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.001040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.001097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.001305 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.001306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.001306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.001417 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.001473 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.001750 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.001836 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:12:35.002133 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:34.993604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46815 (local address 127.19.66.194:50351)
0620 19:12:34.993752 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:34.993787 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:34.994637 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:34.994835 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:34.994838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:34.995074 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:34.995265 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:34.995269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.995901 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.995902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:34.996755 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:34.996758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:34.997182 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:34.997183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:34.997326 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:34.997799 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:34.997809 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:34.999313 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:35.000698 (+ 1385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.000705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.000707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.000975 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.001148 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.001148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.001149 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.001253 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.001506 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:35.001512 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:35.001636 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.001725 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.001877 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 19:12:35.002555 20618 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46815
I20260620 19:12:35.002655 20618 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:35.002821 20618 heartbeater.cc:511] Master 127.19.66.254:46815 requested a full tablet report, sending...
I20260620 19:12:35.003189 20293 ts_manager.cc:194] Registered new tserver with Master: 7ce634ace0c2491897a3eaa9f685c13e (127.19.66.194:41053)
I20260620 19:12:35.003685 20293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:50351
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:35.019488 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:35.096292 20623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:35.096484 20623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:35.096513 20623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:35.097641 20623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:35.097680 20623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:35.097714 20623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:35.099157 20623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46815
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:35.099380 20623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:35.099575 20623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:35.104122 20629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.104161 20628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:35.104131 20631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:35.104369 20623 server_base.cc:1034] running on GCE node
I20260620 19:12:35.104542 20623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:35.104724 20623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:35.105847 20623 hybrid_clock.cc:648] HybridClock initialized: now 1781982755105838 us; error 27 us; skew 500 ppm
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:35.108161 20623 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:35.108559 20623 webserver.cc:460] Webserver started at http://127.19.66.195:37819/ using document root <none> and password file <none>
I20260620 19:12:35.108814 20623 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:35.108891 20623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:35.109010 20623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:35.110002 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-nq8r"
server_key: "17cc5e9188cbfbd79d2428b32939e3ec"
server_key_iv: "d939aad8abc842076a5c28d0b0bae941"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.110301 20623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-nq8r"
server_key: "17cc5e9188cbfbd79d2428b32939e3ec"
server_key_iv: "d939aad8abc842076a5c28d0b0bae941"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.111452 20623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:35.112078 20638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.112222 20623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:35.112298 20623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec"
format_stamp: "Formatted at 2026-06-20 19:12:35 on dist-test-slave-nq8r"
server_key: "17cc5e9188cbfbd79d2428b32939e3ec"
server_key_iv: "d939aad8abc842076a5c28d0b0bae941"
server_key_version: "encryptionkey@0"
I20260620 19:12:35.112352 20623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:35.119601 20623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:35.121219 20623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:35.121343 20623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:35.121528 20623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:35.121793 20623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:35.121825 20623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.121851 20623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:35.121874 20623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:35.127071 20623 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:44649
I20260620 19:12:35.127105 20751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:44649 every 8 connection(s)
I20260620 19:12:35.127383 20623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:35.133353 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20623
I20260620 19:12:35.133445 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:35.133606 19723 external_mini_cluster.cc:1428] Setting key 3de674bba2e1d1fdb70e02990313c9c6
I20260620 19:12:35.135788 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.128018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:46787 (local address 127.19.66.254:46815)
0620 19:12:35.128082 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.128084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.128248 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:12:35.129151 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.129152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.129162 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.129229 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.129827 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.130336 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.131253 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.131445 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.131492 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.133417 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.133418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.133419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.133428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.134812 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.135135 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.135136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.135136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.135181 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.135331 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.135332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.135332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.135423 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.135468 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.135659 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.135724 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:35.135968 20754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.127977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46815 (local address 127.19.66.195:46787)
0620 19:12:35.128134 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.128174 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.129061 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.129285 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.129288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.129537 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:35.129718 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.129722 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.130407 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.130408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.131189 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.131190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.131502 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.131503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.131600 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.132033 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:35.132040 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:35.133331 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:35.134879 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.134885 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.134887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.135085 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.135216 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:35.135217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:35.135217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:35.135294 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:35.135466 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:35.135468 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:35.135570 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.135643 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.135763 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260620 19:12:35.136350 20752 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46815
I20260620 19:12:35.136439 20752 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:35.136593 20752 heartbeater.cc:511] Master 127.19.66.254:46815 requested a full tablet report, sending...
I20260620 19:12:35.136941 20293 ts_manager.cc:194] Registered new tserver with Master: c7ec2d6ef0ab4cc291643426ff1e6fec (127.19.66.195:44649)
I20260620 19:12:35.137285 20293 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:46787
I20260620 19:12:35.145298 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:35.153072 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.147122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55324 (local address 127.19.66.254:46815)
0620 19:12:35.147167 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.147170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.147338 (+ 168us) server_negotiation.cc:407] Connection header received
0620 19:12:35.147404 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.147405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.147415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.147465 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.148089 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.148703 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.149446 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.149644 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.149697 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.150789 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.150790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.150790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.150797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.152053 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.152346 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.152346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.152347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.152400 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.152555 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.152556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.152556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.152694 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.152781 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.152916 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.152984 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:12:35.154954 20293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:35.156414 20293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:35.162109 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:32971 (local address 127.0.0.1:39262)
0620 19:12:35.158350 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.158378 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.158436 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.158781 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.158784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.158786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.158845 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.158846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.159878 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.159879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.161021 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.161023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.161837 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.161913 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.161925 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.161987 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.162035 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260620 19:12:35.162117 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.19.66.193:32971)
0620 19:12:35.158534 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.158539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.158549 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:12:35.158588 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.158591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.158606 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.158682 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.158912 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.159784 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161171 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.161762 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161820 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.161877 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.162004 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.162044 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260620 19:12:35.162257 20762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41053 (local address 127.0.0.1:57450)
0620 19:12:35.158669 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.158678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.158716 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.158965 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.158966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.158967 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.158999 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.159000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.160265 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.160266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.161072 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.161074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.162037 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.162038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.162097 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.162101 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.162143 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.162183 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":19,"thread_start_us":51,"threads_started":1}
I20260620 19:12:35.162411 20764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:44649 (local address 127.0.0.1:38064)
0620 19:12:35.158684 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.158693 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.158737 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.159015 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.159016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.159018 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.159062 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.159063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.160242 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.160243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.161350 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.161352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.162209 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.162210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.162262 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.162265 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.162306 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.162337 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":32,"thread_start_us":61,"threads_started":1}
I20260620 19:12:35.162457 20763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.19.66.194:41053)
0620 19:12:35.158834 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.158837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.158845 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:35.158878 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.158880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.158893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.158964 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.159037 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.160174 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161192 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.161871 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.162003 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.162346 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.162385 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":71,"threads_started":1}
I20260620 19:12:35.162611 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.158635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.19.66.195:44649)
0620 19:12:35.158789 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.158795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.158803 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:35.158838 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.158840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.158856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.158936 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.159130 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.160139 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.161415 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.162078 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.162135 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.162180 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.162501 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.162543 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":60,"threads_started":1}
I20260620 19:12:35.162935 20418 tablet_service.cc:1467] Processing CreateTablet for tablet 92bf92fadf3e4de68456407ba9974b64 (DEFAULT_TABLE table=test-table [id=8bd2f275ba544f098b268f40292847fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:35.162978 20552 tablet_service.cc:1467] Processing CreateTablet for tablet 92bf92fadf3e4de68456407ba9974b64 (DEFAULT_TABLE table=test-table [id=8bd2f275ba544f098b268f40292847fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:35.163103 20686 tablet_service.cc:1467] Processing CreateTablet for tablet 92bf92fadf3e4de68456407ba9974b64 (DEFAULT_TABLE table=test-table [id=8bd2f275ba544f098b268f40292847fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:35.163268 20552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bf92fadf3e4de68456407ba9974b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:35.163268 20418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bf92fadf3e4de68456407ba9974b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:35.163352 20686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bf92fadf3e4de68456407ba9974b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:35.165767 20766 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Bootstrap starting.
I20260620 19:12:35.166546 20766 tablet_bootstrap.cc:654] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:35.166711 20767 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Bootstrap starting.
I20260620 19:12:35.167011 20766 log.cc:824] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:35.167200 20767 tablet_bootstrap.cc:654] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:35.167641 20766 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: No bootstrap required, opened a new log
I20260620 19:12:35.167685 20767 log.cc:824] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:35.167697 20766 ts_tablet_manager.cc:1397] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:35.167979 20768 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Bootstrap starting.
I20260620 19:12:35.168282 20767 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: No bootstrap required, opened a new log
I20260620 19:12:35.168339 20767 ts_tablet_manager.cc:1397] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:35.169090 20766 raft_consensus.cc:348] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.169224 20766 raft_consensus.cc:374] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:35.169255 20766 raft_consensus.cc:729] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ec2d6ef0ab4cc291643426ff1e6fec, State: Initialized, Role: FOLLOWER
I20260620 19:12:35.169378 20766 consensus_queue.cc:260] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.169426 20768 tablet_bootstrap.cc:654] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:35.169620 20766 ts_tablet_manager.cc:1428] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:35.169790 20752 heartbeater.cc:503] Master 127.19.66.254:46815 was elected leader, sending a full tablet report...
I20260620 19:12:35.169957 20768 log.cc:824] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:35.170079 20767 raft_consensus.cc:348] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.170240 20767 raft_consensus.cc:374] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:35.170274 20767 raft_consensus.cc:729] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc5b3d524b5475db6f6c086b2839218, State: Initialized, Role: FOLLOWER
I20260620 19:12:35.170383 20767 consensus_queue.cc:260] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.170620 20484 heartbeater.cc:503] Master 127.19.66.254:46815 was elected leader, sending a full tablet report...
I20260620 19:12:35.170625 20767 ts_tablet_manager.cc:1428] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:35.170900 20768 tablet_bootstrap.cc:492] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: No bootstrap required, opened a new log
I20260620 19:12:35.170967 20768 ts_tablet_manager.cc:1397] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:35.172377 20768 raft_consensus.cc:348] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.172509 20768 raft_consensus.cc:374] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:35.172529 20768 raft_consensus.cc:729] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ce634ace0c2491897a3eaa9f685c13e, State: Initialized, Role: FOLLOWER
I20260620 19:12:35.172631 20768 consensus_queue.cc:260] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.172892 20768 ts_tablet_manager.cc:1428] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:35.172906 20618 heartbeater.cc:503] Master 127.19.66.254:46815 was elected leader, sending a full tablet report...
W20260620 19:12:35.243525 20619 tablet.cc:2369] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:35.323405 20774 raft_consensus.cc:482] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:35.323526 20774 raft_consensus.cc:504] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.323817 20774 leader_election.cc:288] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adc5b3d524b5475db6f6c086b2839218 (127.19.66.193:32971), c7ec2d6ef0ab4cc291643426ff1e6fec (127.19.66.195:44649)
I20260620 19:12:35.326500 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.324095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:42383 (local address 127.19.66.193:32971)
0620 19:12:35.324143 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.324145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.324157 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:35.324194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.324195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.324204 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.324259 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.324399 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.324923 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.325828 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.326197 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.326257 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.326266 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.326401 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.326436 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:12:35.326521 20765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.324239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50861 (local address 127.19.66.195:44649)
0620 19:12:35.324281 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.324282 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.324292 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:12:35.324349 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.324350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.324357 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.324415 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:35.324546 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.325013 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.325870 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.326222 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.326279 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.326287 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.326423 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.326456 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:35.326534 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.324012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:32971 (local address 127.19.66.194:42383)
0620 19:12:35.324071 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.324082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.324145 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.324264 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.324265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.324267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.324334 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.324336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.324993 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.324994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.325739 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.325741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.326268 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.326269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.326341 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.326344 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.326389 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.326429 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":7}
I20260620 19:12:35.326539 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.324118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:44649 (local address 127.19.66.194:50861)
0620 19:12:35.324258 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:35.324266 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:35.324302 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:35.324426 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:35.324427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:35.324429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:35.324491 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.324493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.325085 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.325086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.325792 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:35.325793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.326294 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.326294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:35.326360 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.326363 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:35.326408 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:35.326444 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260620 19:12:35.326830 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bf92fadf3e4de68456407ba9974b64" candidate_uuid: "7ce634ace0c2491897a3eaa9f685c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" is_pre_election: true
I20260620 19:12:35.326874 20438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bf92fadf3e4de68456407ba9974b64" candidate_uuid: "7ce634ace0c2491897a3eaa9f685c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc5b3d524b5475db6f6c086b2839218" is_pre_election: true
I20260620 19:12:35.326974 20706 raft_consensus.cc:2454] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce634ace0c2491897a3eaa9f685c13e in term 0.
I20260620 19:12:35.327018 20438 raft_consensus.cc:2454] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ce634ace0c2491897a3eaa9f685c13e in term 0.
I20260620 19:12:35.327200 20508 leader_election.cc:302] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce634ace0c2491897a3eaa9f685c13e, c7ec2d6ef0ab4cc291643426ff1e6fec; no voters:
I20260620 19:12:35.327322 20774 raft_consensus.cc:2781] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:35.327373 20774 raft_consensus.cc:482] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:35.327409 20774 raft_consensus.cc:3037] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:35.328044 20774 raft_consensus.cc:504] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.328142 20774 leader_election.cc:288] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [CANDIDATE]: Term 1 election: Requested vote from peers adc5b3d524b5475db6f6c086b2839218 (127.19.66.193:32971), c7ec2d6ef0ab4cc291643426ff1e6fec (127.19.66.195:44649)
I20260620 19:12:35.328336 20706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bf92fadf3e4de68456407ba9974b64" candidate_uuid: "7ce634ace0c2491897a3eaa9f685c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec"
I20260620 19:12:35.328321 20438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bf92fadf3e4de68456407ba9974b64" candidate_uuid: "7ce634ace0c2491897a3eaa9f685c13e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc5b3d524b5475db6f6c086b2839218"
I20260620 19:12:35.328399 20706 raft_consensus.cc:3037] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:35.328402 20438 raft_consensus.cc:3037] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:35.329079 20706 raft_consensus.cc:2454] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce634ace0c2491897a3eaa9f685c13e in term 1.
I20260620 19:12:35.329176 20438 raft_consensus.cc:2454] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ce634ace0c2491897a3eaa9f685c13e in term 1.
I20260620 19:12:35.329224 20508 leader_election.cc:302] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ce634ace0c2491897a3eaa9f685c13e, c7ec2d6ef0ab4cc291643426ff1e6fec; no voters:
I20260620 19:12:35.329334 20774 raft_consensus.cc:2781] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:35.329473 20774 raft_consensus.cc:686] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [term 1 LEADER]: Becoming Leader. State: Replica: 7ce634ace0c2491897a3eaa9f685c13e, State: Running, Role: LEADER
I20260620 19:12:35.329561 20774 consensus_queue.cc:237] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } }
I20260620 19:12:35.330349 20293 catalog_manager.cc:5495] T 92bf92fadf3e4de68456407ba9974b64 P 7ce634ace0c2491897a3eaa9f685c13e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ce634ace0c2491897a3eaa9f685c13e (127.19.66.194). New cstate: current_term: 1 leader_uuid: "7ce634ace0c2491897a3eaa9f685c13e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "adc5b3d524b5475db6f6c086b2839218" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ce634ace0c2491897a3eaa9f685c13e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7ec2d6ef0ab4cc291643426ff1e6fec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 44649 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:35.344709 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.338108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.19.66.193:32971)
0620 19:12:35.338155 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.338157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.338260 (+ 103us) server_negotiation.cc:407] Connection header received
0620 19:12:35.338317 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.338317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.338327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.338371 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.338886 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.339347 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.340010 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.340191 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.340243 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.342389 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.342401 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.342401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.342408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.343654 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.343957 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.343961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.343961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.344011 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.344169 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.344170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.344170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.344348 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.344486 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.344546 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.344621 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260620 19:12:35.356675 20485 tablet.cc:2369] T 92bf92fadf3e4de68456407ba9974b64 P adc5b3d524b5475db6f6c086b2839218: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:35.365036 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.359039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39268 (local address 127.19.66.193:32971)
0620 19:12:35.359095 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.359096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.359166 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:12:35.359209 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.359210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.359218 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.359262 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.359720 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.360170 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.360879 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.361055 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.361100 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.363010 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.363011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.363012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.363017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.364166 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.364457 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.364458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.364458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.364503 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.364630 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.364630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.364630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.364704 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.364765 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.364914 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.364970 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260620 19:12:35.377854 20753 tablet.cc:2369] T 92bf92fadf3e4de68456407ba9974b64 P c7ec2d6ef0ab4cc291643426ff1e6fec: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:35.384600 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:35.378457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39282 (local address 127.19.66.193:32971)
0620 19:12:35.378520 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:35.378523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:35.378615 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:12:35.378677 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:35.378678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:35.378688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:35.378735 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:35.379243 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.379705 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.380393 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:35.380581 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:35.380634 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:35.382618 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:35.382619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:35.382619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:35.382627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:35.383671 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.383958 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.383959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.383960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.384009 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:35.384161 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:35.384162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:35.384162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:35.384251 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:35.384331 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:35.384433 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 19:12:35.384512 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 19:12:35.385038 20398 tablet_service.cc:2908] Rejecting scan request for tablet 92bf92fadf3e4de68456407ba9974b64: Uninitialized: clean time has not yet been initialized
I20260620 19:12:35.385732 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20355
I20260620 19:12:35.390859 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20489
I20260620 19:12:35.394676 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20623
I20260620 19:12:35.398267 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20261
2026-06-20T19:12:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1867 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20790](info): set up 2 sockets
Jun 20 19:12:35 dist-test-slave-nq8r krb5kdc[20790](info): commencing operation
krb5kdc: starting...
W20260620 19:12:36.344157 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:36.449270 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35979
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:36.527052 20803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.527271 20803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.527300 20803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.528470 20803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:36.528509 20803 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:12:36.528527 20803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.528542 20803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.528558 20803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:36.528574 20803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:36.530056 20803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35979
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:36.530296 20803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.530507 20803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.534989 20811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.535077 20809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.534991 20808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.535257 20803 server_base.cc:1034] running on GCE node
I20260620 19:12:36.535466 20803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.535727 20803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.536880 20803 hybrid_clock.cc:648] HybridClock initialized: now 1781982756536864 us; error 30 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.542415 20803 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:36.542804 20803 webserver.cc:460] Webserver started at http://127.19.66.254:35107/ using document root <none> and password file <none>
I20260620 19:12:36.542989 20803 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.543045 20803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.543144 20803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:36.544128 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f3443165e33d4d82ac766a8f1271cf7b"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "cf5391f0cc3076c61af0a9b3745d4b28"
server_key_iv: "25f0c0ffc2ca132a905069165f556519"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.544417 20803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f3443165e33d4d82ac766a8f1271cf7b"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "cf5391f0cc3076c61af0a9b3745d4b28"
server_key_iv: "25f0c0ffc2ca132a905069165f556519"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.545656 20803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:36.546262 20818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.546435 20803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:36.546509 20803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f3443165e33d4d82ac766a8f1271cf7b"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "cf5391f0cc3076c61af0a9b3745d4b28"
server_key_iv: "25f0c0ffc2ca132a905069165f556519"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.546574 20803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.559770 20803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.561868 20803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.561997 20803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.566185 20803 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:35979
I20260620 19:12:36.566263 20880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:35979 every 8 connection(s)
I20260620 19:12:36.566553 20803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:36.567204 20881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:36.569410 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Bootstrap starting.
I20260620 19:12:36.569926 20881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:36.570145 20881 log.cc:824] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:36.570751 20881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: No bootstrap required, opened a new log
I20260620 19:12:36.572767 20881 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } }
I20260620 19:12:36.572943 20881 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:36.572969 20881 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3443165e33d4d82ac766a8f1271cf7b, State: Initialized, Role: FOLLOWER
I20260620 19:12:36.573107 20881 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } }
I20260620 19:12:36.573163 20881 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:36.573184 20881 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:36.573233 20881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:36.573515 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20803
I20260620 19:12:36.573604 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:36.573750 19723 external_mini_cluster.cc:1428] Setting key e579bbdae61a5cec30da83995e776102
I20260620 19:12:36.573870 20881 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } }
I20260620 19:12:36.573968 20881 leader_election.cc:302] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3443165e33d4d82ac766a8f1271cf7b; no voters:
I20260620 19:12:36.574117 20881 leader_election.cc:288] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:36.574205 20884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:36.574358 20881 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:36.574517 20884 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [term 1 LEADER]: Becoming Leader. State: Replica: f3443165e33d4d82ac766a8f1271cf7b, State: Running, Role: LEADER
I20260620 19:12:36.574602 20884 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } }
I20260620 19:12:36.574930 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3443165e33d4d82ac766a8f1271cf7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } } }
I20260620 19:12:36.575007 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:36.575121 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3443165e33d4d82ac766a8f1271cf7b. Latest consensus state: current_term: 1 leader_uuid: "f3443165e33d4d82ac766a8f1271cf7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3443165e33d4d82ac766a8f1271cf7b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35979 } } }
I20260620 19:12:36.575169 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:36.575326 20895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:36.576013 20895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:36.577622 20895 catalog_manager.cc:1347] Generated new cluster ID: 3fdcb2329f784c8b8abf7a181232c579
I20260620 19:12:36.577675 20895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:36.586596 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.574375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.19.66.254:35979)
0620 19:12:36.576053 (+ 1678us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.576063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.576095 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:12:36.576207 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.576214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.576269 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.576486 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.577303 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.578638 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.579723 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.580370 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.580484 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.583188 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.583192 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.583193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.583204 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.585064 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.585445 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.585448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.585448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.585553 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.585735 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.585736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.585736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.586022 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.586157 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.586249 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.586378 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1631,"thread_start_us":56,"threads_started":1}
I20260620 19:12:36.595314 20895 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:36.595882 20895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:36.599968 20895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Generated new TSK 0
I20260620 19:12:36.600129 20895 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:36.625100 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35979
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:36.707639 20907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.707830 20907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.707852 20907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.708999 20907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.709036 20907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.709059 20907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:36.709087 20907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:36.710477 20907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:36.710680 20907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.710886 20907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.715384 20913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.715396 20912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.715396 20915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.715601 20907 server_base.cc:1034] running on GCE node
I20260620 19:12:36.715847 20907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.716063 20907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.717193 20907 hybrid_clock.cc:648] HybridClock initialized: now 1781982756717178 us; error 26 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.719457 20907 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:36.719851 20907 webserver.cc:460] Webserver started at http://127.19.66.193:46537/ using document root <none> and password file <none>
I20260620 19:12:36.720048 20907 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.720095 20907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.720216 20907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:36.721240 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "0cabec436f2644bba9db53aca9c6750f"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "935484f23f5dca896174a3759a27382d"
server_key_iv: "1fb6236123b8026ee96d934a62e51ed3"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.721552 20907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "0cabec436f2644bba9db53aca9c6750f"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "935484f23f5dca896174a3759a27382d"
server_key_iv: "1fb6236123b8026ee96d934a62e51ed3"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.722764 20907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:36.723414 20922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.723557 20907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:36.723635 20907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "0cabec436f2644bba9db53aca9c6750f"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "935484f23f5dca896174a3759a27382d"
server_key_iv: "1fb6236123b8026ee96d934a62e51ed3"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.723692 20907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.740226 20907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.742280 20907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.742403 20907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.742898 20907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:36.742937 20907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.742966 20907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:36.742987 20907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:36.749446 20907 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:34099
I20260620 19:12:36.749536 21037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:34099 every 8 connection(s)
I20260620 19:12:36.749814 20907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:36.752436 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.743223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:49403 (local address 127.19.66.254:35979)
0620 19:12:36.743286 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.743288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.743485 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:12:36.743940 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.743942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.743955 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.744027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.744615 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.745215 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.746159 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.746404 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.746459 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.749077 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.749079 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.749079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.749091 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.751061 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.751489 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.751491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.751491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.751577 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.751828 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.751829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.751830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.751950 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.752006 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.752254 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.752349 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:12:36.752568 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.743220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.193:49403)
0620 19:12:36.743363 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.743402 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.743847 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.744091 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.744096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.744351 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.744526 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.744531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.745298 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.745299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.746080 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.746082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.746472 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.746473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.746579 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.747421 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.747431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.748977 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.751136 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.751143 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.751145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.751427 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.751641 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.751642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.751642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.751769 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.752006 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.752013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.752139 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.752225 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.752370 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260620 19:12:36.753036 21038 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35979
I20260620 19:12:36.753114 21038 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:36.753284 21038 heartbeater.cc:511] Master 127.19.66.254:35979 requested a full tablet report, sending...
I20260620 19:12:36.753813 20834 ts_manager.cc:194] Registered new tserver with Master: 0cabec436f2644bba9db53aca9c6750f (127.19.66.193:34099)
I20260620 19:12:36.754431 20834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:49403
I20260620 19:12:36.758772 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.753941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.193:52639)
0620 19:12:36.754055 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.754065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.754113 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.754216 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.754216 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.754376 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.754445 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.754447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.755275 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.755275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.755789 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.755790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.756085 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.756086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.756134 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.756446 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.756450 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.756686 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.758100 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.758101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.758102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.758247 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.758383 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.758384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.758384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.758425 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.758569 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.758570 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.758607 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.758649 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.758712 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260620 19:12:36.758775 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.754081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:52639 (local address 127.19.66.254:35979)
0620 19:12:36.754113 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.754115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.754123 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:36.754162 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.754163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.754171 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.754212 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.754511 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.755194 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.755850 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.756039 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.756082 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.756751 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.756752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.756753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.756758 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.758028 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.758300 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.758300 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.758300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.758345 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.758455 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.758456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.758456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.758537 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.758576 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.758665 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.758720 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:12:36.759865 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 20907
I20260620 19:12:36.759963 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:36.760115 19723 external_mini_cluster.cc:1428] Setting key b97eaed81577e0a34b5e895fb00d1207
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:36.778890 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35979
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:36.854339 21047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:36.854518 21047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:36.854539 21047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:36.855638 21047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:36.855676 21047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:36.855695 21047 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:36.855739 21047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:36.857121 21047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:36.857314 21047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:36.857508 21047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:36.861999 21052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.862028 21055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:36.862018 21053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:36.862118 21047 server_base.cc:1034] running on GCE node
I20260620 19:12:36.862558 21047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:36.862764 21047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:36.863894 21047 hybrid_clock.cc:648] HybridClock initialized: now 1781982756863881 us; error 27 us; skew 500 ppm
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:36.866195 21047 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:36.866582 21047 webserver.cc:460] Webserver started at http://127.19.66.194:42167/ using document root <none> and password file <none>
I20260620 19:12:36.866767 21047 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:36.866812 21047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:36.866930 21047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:36.867852 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "019bcc8068f86fce8d714fca0837aa77"
server_key_iv: "a6b08638730802340d36520c624dac30"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.868160 21047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "019bcc8068f86fce8d714fca0837aa77"
server_key_iv: "a6b08638730802340d36520c624dac30"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.869350 21047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:36.869961 21062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.870100 21047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:36.870175 21047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
format_stamp: "Formatted at 2026-06-20 19:12:36 on dist-test-slave-nq8r"
server_key: "019bcc8068f86fce8d714fca0837aa77"
server_key_iv: "a6b08638730802340d36520c624dac30"
server_key_version: "encryptionkey@0"
I20260620 19:12:36.870231 21047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:36.894937 21047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:36.897188 21047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:36.897336 21047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:36.897827 21047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:36.897866 21047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:36.897895 21047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:36.897912 21047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:12:36 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:36.904217 21047 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:36761
I20260620 19:12:36.904300 21177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:36761 every 8 connection(s)
I20260620 19:12:36.904584 21047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:36.906625 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.898283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:57205 (local address 127.19.66.254:35979)
0620 19:12:36.898349 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.898351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.898556 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:12:36.899074 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.899076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.899089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.899161 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.899963 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.900699 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.901792 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.902092 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.902158 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.904307 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.904309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.904309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.904317 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.905636 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.905972 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.905973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.905973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.906019 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.906174 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.906174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.906174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.906265 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.906309 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.906505 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.906563 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:12:36.906790 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.898288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.194:57205)
0620 19:12:36.898439 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.898474 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.898952 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.899233 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.899238 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.899583 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.899856 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.899863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.900814 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.900815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.901719 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.901721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.902190 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.902191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.902332 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.902811 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.902819 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.904201 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.905700 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.905703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.905705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.905908 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.906054 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.906054 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.906055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.906136 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.906306 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.906309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.906415 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.906491 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.906606 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":88,"threads_started":1}
I20260620 19:12:36.907220 21178 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35979
I20260620 19:12:36.907301 21178 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:36.907487 21178 heartbeater.cc:511] Master 127.19.66.254:35979 requested a full tablet report, sending...
I20260620 19:12:36.907828 20835 ts_manager.cc:194] Registered new tserver with Master: 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761)
I20260620 19:12:36.908294 20835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:57205
I20260620 19:12:36.912756 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.908193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:58685 (local address 127.19.66.254:35979)
0620 19:12:36.908238 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:36.908240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:36.908358 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:12:36.908456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:36.908456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:36.908465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:36.908503 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:36.908790 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.909293 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.909936 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.910130 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.910180 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.910902 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:36.910903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:36.910904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:36.910910 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:36.911850 (+ 940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.912160 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.912161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.912161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.912215 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:36.912377 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:36.912377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:36.912378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:36.912502 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:36.912547 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:36.912643 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:12:36.912699 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:12:36.912806 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:36.908129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.194:58685)
0620 19:12:36.908291 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:36.908300 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:36.908359 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:36.908513 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:36.908513 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:36.908676 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:36.908727 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.908729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.909375 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.909376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.909870 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:36.909871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:36.910183 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:36.910183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:36.910253 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:36.910584 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:36.910587 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:36.910829 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:36.911921 (+ 1092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.911922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.911923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.912088 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.912272 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:36.912273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:36.912273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:36.912338 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:36.912544 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:36.912544 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:36.912578 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:12:36.912619 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:12:36.912710 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260620 19:12:36.912811 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21047
I20260620 19:12:36.912912 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:36.913081 19723 external_mini_cluster.cc:1428] Setting key 2bb1e6aa42d245e4a75b65e0221d805d
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:36.934533 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35979
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:33087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:37.013363 21187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:37.013553 21187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:37.013582 21187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:37.014684 21187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:37.014721 21187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:37.014748 21187 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:37.014771 21187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:37.016145 21187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:33087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:37.016368 21187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:37.016556 21187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:37.021504 21187 server_base.cc:1034] running on GCE node
W20260620 19:12:37.021499 21195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.021512 21193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:37.021546 21192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:37.021958 21187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:37.022204 21187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:37.023376 21187 hybrid_clock.cc:648] HybridClock initialized: now 1781982757023361 us; error 28 us; skew 500 ppm
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:37.025758 21187 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:37.026188 21187 webserver.cc:460] Webserver started at http://127.19.66.195:40647/ using document root <none> and password file <none>
I20260620 19:12:37.026378 21187 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:37.026456 21187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:37.026577 21187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:37.027580 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "a53cde13e4b24fa8bb0d54e510014eca"
format_stamp: "Formatted at 2026-06-20 19:12:37 on dist-test-slave-nq8r"
server_key: "3a765f9edcacddc29ce874c7d458e094"
server_key_iv: "57c408756976a12513e154084a2a9942"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.027895 21187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "a53cde13e4b24fa8bb0d54e510014eca"
format_stamp: "Formatted at 2026-06-20 19:12:37 on dist-test-slave-nq8r"
server_key: "3a765f9edcacddc29ce874c7d458e094"
server_key_iv: "57c408756976a12513e154084a2a9942"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.029112 21187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:37.029762 21202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.029902 21187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:37.029971 21187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "a53cde13e4b24fa8bb0d54e510014eca"
format_stamp: "Formatted at 2026-06-20 19:12:37 on dist-test-slave-nq8r"
server_key: "3a765f9edcacddc29ce874c7d458e094"
server_key_iv: "57c408756976a12513e154084a2a9942"
server_key_version: "encryptionkey@0"
I20260620 19:12:37.030032 21187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:37.042691 21187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:37.044303 21187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:37.044418 21187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:37.044909 21187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:37.044945 21187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.044965 21187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:37.044982 21187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.051056 21187 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:40653
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:37.051115 21317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:40653 every 8 connection(s)
I20260620 19:12:37.051417 21187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:37.053793 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.045268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:38133 (local address 127.19.66.254:35979)
0620 19:12:37.045329 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.045332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.045504 (+ 172us) server_negotiation.cc:407] Connection header received
0620 19:12:37.045943 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.045945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.045957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.046035 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.046763 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.047502 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.048950 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.049148 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.049205 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.051591 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.051593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.051593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.051602 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.052701 (+ 1099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.053083 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.053084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.053084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.053132 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.053337 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.053337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.053338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.053440 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.053485 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.053673 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.053733 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:12:37.053936 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.045238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.195:38133)
0620 19:12:37.045383 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.045421 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.045856 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.046086 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.046089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.046403 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:37.046668 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.046674 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.047582 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.047583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.048801 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.048803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.049227 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.049227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.049375 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.049827 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:37.049835 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:37.051508 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:37.052761 (+ 1253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.052766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.052767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.053024 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.053187 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.053188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.053188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.053300 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.053485 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:37.053488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:37.053587 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.053661 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.053768 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260620 19:12:37.054386 21318 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35979
I20260620 19:12:37.054462 21318 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:37.054611 21318 heartbeater.cc:511] Master 127.19.66.254:35979 requested a full tablet report, sending...
I20260620 19:12:37.054961 20834 ts_manager.cc:194] Registered new tserver with Master: a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653)
I20260620 19:12:37.055418 20834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:38133
I20260620 19:12:37.058406 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21187
I20260620 19:12:37.058501 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:37.058648 19723 external_mini_cluster.cc:1428] Setting key 105c75b4f686f7e8b6c25eedfe72cabe
I20260620 19:12:37.059407 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:37.060544 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.055269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:54863 (local address 127.19.66.254:35979)
0620 19:12:37.055308 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.055310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.055369 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:12:37.055433 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.055434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.055441 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.055483 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.055835 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.056404 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.057102 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.057272 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.057313 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.058113 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.058113 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.058114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.058117 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.059573 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.059897 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.059898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.059898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.059954 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.060138 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.060139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.060139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.060246 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.060294 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.060435 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.060486 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:37.060587 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.055206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.19.66.195:54863)
0620 19:12:37.055317 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.055327 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.055380 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.055517 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.055518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.055693 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:37.055769 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.055771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.056485 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.056486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.057043 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.057044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.057354 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.057355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.057430 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.057804 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:37.057809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:37.058052 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:37.059634 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.059635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.059636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.059825 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.060003 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.060004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.060004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.060063 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.060297 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:37.060298 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:37.060341 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.060408 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.060497 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:37.080415 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.074029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.19.66.254:35979)
0620 19:12:37.074069 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.074071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.074079 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:37.074152 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.074153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.074164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.074208 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.074771 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.075291 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.076042 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.076233 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.076288 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.078375 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.078376 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.078376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.078383 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.079617 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.079900 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.079901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.079901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.079949 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.080068 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.080068 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.080069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.080156 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.080219 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.080286 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.080343 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:37.081950 20834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:37.083454 20834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:37.089708 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.19.66.194:36761)
0620 19:12:37.085917 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.085921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.086133 (+ 212us) server_negotiation.cc:407] Connection header received
0620 19:12:37.086248 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.086252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.086273 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.086342 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.086554 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087310 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.088531 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089304 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089374 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089429 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.089571 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.089608 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":116,"threads_started":1}
I20260620 19:12:37.089685 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36761 (local address 127.0.0.1:57446)
0620 19:12:37.086035 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.086078 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.086151 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.086417 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.086420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.086422 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.086488 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.086489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087382 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.087384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.088448 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.088450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089382 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.089467 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089483 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.089552 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.089604 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":82,"threads_started":1}
I20260620 19:12:37.089951 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.19.66.193:34099)
0620 19:12:37.085874 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.085880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.086443 (+ 563us) server_negotiation.cc:407] Connection header received
0620 19:12:37.086486 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.086488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.086502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.086568 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.086731 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087796 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.088893 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089406 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089470 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089524 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.089768 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.089862 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":81,"threads_started":1}
I20260620 19:12:37.090081 21332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40653 (local address 127.0.0.1:51364)
0620 19:12:37.086264 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.086272 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.086315 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.086529 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.086530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.086533 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.086578 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.086580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087838 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.087839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.088963 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.088966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089882 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.089929 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089932 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.089976 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.090014 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":78,"thread_start_us":57,"threads_started":1}
I20260620 19:12:37.090126 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.19.66.195:40653)
0620 19:12:37.086007 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.086010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.086317 (+ 307us) server_negotiation.cc:407] Connection header received
0620 19:12:37.086382 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.086384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.086397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.086459 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.086644 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087748 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089050 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089740 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089803 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089852 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.090002 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.090037 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260620 19:12:37.090559 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.085502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:34099 (local address 127.0.0.1:55428)
0620 19:12:37.086378 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.086386 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.086427 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.086631 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.086632 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.086634 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.086661 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.086662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.087889 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.087890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.088809 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.088811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.089555 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.089556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.089607 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.089610 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.089653 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.089682 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":24,"thread_start_us":50,"threads_started":1}
I20260620 19:12:37.090796 21112 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 (DEFAULT_TABLE table=test-table [id=8e0f4106c92c478593dd306c228c724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.091150 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeca1e103fd4d7983e5a49190232bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.091157 20972 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 (DEFAULT_TABLE table=test-table [id=8e0f4106c92c478593dd306c228c724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.091604 20972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeca1e103fd4d7983e5a49190232bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.094049 21338 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Bootstrap starting.
I20260620 19:12:37.094329 21252 tablet_service.cc:1467] Processing CreateTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 (DEFAULT_TABLE table=test-table [id=8e0f4106c92c478593dd306c228c724c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.094599 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aeca1e103fd4d7983e5a49190232bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.094645 21339 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Bootstrap starting.
I20260620 19:12:37.094702 21338 tablet_bootstrap.cc:654] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.095024 21338 log.cc:824] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.095253 21339 tablet_bootstrap.cc:654] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.095558 21339 log.cc:824] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.095790 21338 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: No bootstrap required, opened a new log
I20260620 19:12:37.095858 21338 ts_tablet_manager.cc:1397] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:37.096387 21339 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: No bootstrap required, opened a new log
I20260620 19:12:37.096447 21339 ts_tablet_manager.cc:1397] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:37.097303 21342 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Bootstrap starting.
I20260620 19:12:37.097841 21338 raft_consensus.cc:348] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.098006 21338 raft_consensus.cc:374] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.098040 21338 raft_consensus.cc:729] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479813a51ef44e2b9b2f869ed3ed95e5, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.098060 21342 tablet_bootstrap.cc:654] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.098179 21338 consensus_queue.cc:260] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.098354 21342 log.cc:824] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:37.098307 21339 raft_consensus.cc:348] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.098451 21338 ts_tablet_manager.cc:1428] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:37.098459 21339 raft_consensus.cc:374] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.098491 21339 raft_consensus.cc:729] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cabec436f2644bba9db53aca9c6750f, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.098610 21178 heartbeater.cc:503] Master 127.19.66.254:35979 was elected leader, sending a full tablet report...
I20260620 19:12:37.098603 21339 consensus_queue.cc:260] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.098865 21339 ts_tablet_manager.cc:1428] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:37.098984 21038 heartbeater.cc:503] Master 127.19.66.254:35979 was elected leader, sending a full tablet report...
I20260620 19:12:37.099165 21342 tablet_bootstrap.cc:492] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: No bootstrap required, opened a new log
I20260620 19:12:37.099229 21342 ts_tablet_manager.cc:1397] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:37.100548 21342 raft_consensus.cc:348] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.100667 21342 raft_consensus.cc:374] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.100705 21342 raft_consensus.cc:729] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53cde13e4b24fa8bb0d54e510014eca, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.100812 21342 consensus_queue.cc:260] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.101068 21342 ts_tablet_manager.cc:1428] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:37.101100 21318 heartbeater.cc:503] Master 127.19.66.254:35979 was elected leader, sending a full tablet report...
W20260620 19:12:37.155179 21179 tablet.cc:2369] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:37.159075 21345 raft_consensus.cc:482] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:37.159197 21345 raft_consensus.cc:504] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.159534 21345 leader_election.cc:288] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653), 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761)
I20260620 19:12:37.162263 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.159693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40653 (local address 127.19.66.193:59975)
0620 19:12:37.159748 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.159760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.159798 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.159994 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.159996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.159999 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.160077 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.160079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.160716 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.160717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.161526 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.161528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.162017 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.162018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.162102 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.162104 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.162150 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.162188 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":71}
I20260620 19:12:37.162253 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.159796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:33183 (local address 127.19.66.194:36761)
0620 19:12:37.159841 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.159843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.159945 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:12:37.160018 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.160019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.160028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.160091 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.160213 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.160673 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.161594 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.161954 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.162015 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.162023 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.162155 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.162188 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:12:37.162259 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.159812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:59975 (local address 127.19.66.195:40653)
0620 19:12:37.159872 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.159874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.159882 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:37.159906 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.159907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.159917 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.159985 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.160144 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.160644 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.161613 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.161949 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.162008 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.162017 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.162164 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.162199 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:37.162276 21347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.159712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36761 (local address 127.19.66.193:33183)
0620 19:12:37.159883 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.159892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.159953 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.160095 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.160096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.160097 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.160164 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.160165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.160745 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.160746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.161511 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.161512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.162028 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.162028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.162094 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.162097 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.162140 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.162182 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
I20260620 19:12:37.162608 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeca1e103fd4d7983e5a49190232bc3" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53cde13e4b24fa8bb0d54e510014eca" is_pre_election: true
I20260620 19:12:37.162662 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeca1e103fd4d7983e5a49190232bc3" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" is_pre_election: true
I20260620 19:12:37.162745 21272 raft_consensus.cc:2454] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 0.
I20260620 19:12:37.162803 21132 raft_consensus.cc:2454] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 0.
I20260620 19:12:37.162988 20924 leader_election.cc:302] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cabec436f2644bba9db53aca9c6750f, a53cde13e4b24fa8bb0d54e510014eca; no voters:
I20260620 19:12:37.163112 21345 raft_consensus.cc:2781] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:37.163166 21345 raft_consensus.cc:482] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:37.163183 21345 raft_consensus.cc:3037] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.164041 21345 raft_consensus.cc:504] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.164148 21345 leader_election.cc:288] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 election: Requested vote from peers a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653), 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761)
I20260620 19:12:37.164291 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeca1e103fd4d7983e5a49190232bc3" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53cde13e4b24fa8bb0d54e510014eca"
I20260620 19:12:37.164335 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aeca1e103fd4d7983e5a49190232bc3" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
I20260620 19:12:37.164373 21272 raft_consensus.cc:3037] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.164390 21132 raft_consensus.cc:3037] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.165083 21132 raft_consensus.cc:2454] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 1.
I20260620 19:12:37.165097 21272 raft_consensus.cc:2454] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 1.
I20260620 19:12:37.165223 20924 leader_election.cc:302] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cabec436f2644bba9db53aca9c6750f, a53cde13e4b24fa8bb0d54e510014eca; no voters:
I20260620 19:12:37.165313 21345 raft_consensus.cc:2781] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:37.165452 21345 raft_consensus.cc:686] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 1 LEADER]: Becoming Leader. State: Replica: 0cabec436f2644bba9db53aca9c6750f, State: Running, Role: LEADER
I20260620 19:12:37.165550 21345 consensus_queue.cc:237] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } }
I20260620 19:12:37.166299 20835 catalog_manager.cc:5495] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cabec436f2644bba9db53aca9c6750f (127.19.66.193). New cstate: current_term: 1 leader_uuid: "0cabec436f2644bba9db53aca9c6750f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } health_report { overall_health: UNKNOWN } } }
I20260620 19:12:37.180558 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.178279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.19.66.193:34099)
0620 19:12:37.178320 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.178321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.178347 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:12:37.178432 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.178433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.178452 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.178505 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.178631 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.179079 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.179830 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.180013 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.180066 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.180248 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.180337 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.180402 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.180454 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.180474 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:37.182135 21132 raft_consensus.cc:1261] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Refusing update from remote peer 0cabec436f2644bba9db53aca9c6750f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.182170 21272 raft_consensus.cc:1261] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Refusing update from remote peer 0cabec436f2644bba9db53aca9c6750f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.182428 21348 consensus_queue.cc:1035] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Connected to new peer: Peer: permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.182535 21345 consensus_queue.cc:1035] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.184659 21351 mvcc.cc:205] Tried to move back new op lower bound from 7299001373415628800 to 7299001373350592512. Current Snapshot: MvccSnapshot[applied={T|T < 7299001373415628800}]
I20260620 19:12:37.185230 21353 mvcc.cc:205] Tried to move back new op lower bound from 7299001373415628800 to 7299001373350592512. Current Snapshot: MvccSnapshot[applied={T|T < 7299001373415628800}]
I20260620 19:12:37.185456 21352 mvcc.cc:205] Tried to move back new op lower bound from 7299001373415628800 to 7299001373350592512. Current Snapshot: MvccSnapshot[applied={T|T < 7299001373415628800}]
I20260620 19:12:37.187898 20834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.187978 20834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.188619 20834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Sending DeleteTablet for 3 replicas of tablet 4aeca1e103fd4d7983e5a49190232bc3
I20260620 19:12:37.188922 20972 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55428
I20260620 19:12:37.188936 21252 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:51364
I20260620 19:12:37.188973 21112 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aeca1e103fd4d7983e5a49190232bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:57446
I20260620 19:12:37.189366 21363 tablet_replica.cc:330] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: stopping tablet replica
I20260620 19:12:37.189465 21363 raft_consensus.cc:2229] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.189440 20834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:37.189566 20834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:37.189574 21363 raft_consensus.cc:2258] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.190088 21363 ts_tablet_manager.cc:1905] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.190660 21252 tablet_service.cc:1467] Processing CreateTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (DEFAULT_TABLE table=test-table [id=1e8137f62177445794db9ba7c5ce0dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.190655 20972 tablet_service.cc:1467] Processing CreateTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (DEFAULT_TABLE table=test-table [id=1e8137f62177445794db9ba7c5ce0dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.190773 20972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.190773 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.192059 21339 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Bootstrap starting.
I20260620 19:12:37.192437 21339 tablet_bootstrap.cc:654] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.192538 21342 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Bootstrap starting.
I20260620 19:12:37.192941 21342 tablet_bootstrap.cc:654] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.193068 21112 tablet_service.cc:1467] Processing CreateTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (DEFAULT_TABLE table=test-table [id=1e8137f62177445794db9ba7c5ce0dd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:37.193174 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.193178 21339 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: No bootstrap required, opened a new log
I20260620 19:12:37.193216 21339 ts_tablet_manager.cc:1397] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:37.193378 21339 raft_consensus.cc:348] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.193450 21339 raft_consensus.cc:374] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.193468 21339 raft_consensus.cc:729] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cabec436f2644bba9db53aca9c6750f, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.193511 21339 consensus_queue.cc:260] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.193569 21342 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: No bootstrap required, opened a new log
I20260620 19:12:37.193586 21339 ts_tablet_manager.cc:1428] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:37.193608 21342 ts_tablet_manager.cc:1397] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:37.193739 21342 raft_consensus.cc:348] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.193796 21342 raft_consensus.cc:374] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.193812 21342 raft_consensus.cc:729] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53cde13e4b24fa8bb0d54e510014eca, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.193848 21342 consensus_queue.cc:260] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.193926 21342 ts_tablet_manager.cc:1428] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.194453 21338 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Bootstrap starting.
I20260620 19:12:37.194779 21338 tablet_bootstrap.cc:654] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.195271 21338 tablet_bootstrap.cc:492] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: No bootstrap required, opened a new log
I20260620 19:12:37.195309 21338 ts_tablet_manager.cc:1397] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:37.195439 21338 raft_consensus.cc:348] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.195501 21338 raft_consensus.cc:374] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.195518 21338 raft_consensus.cc:729] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479813a51ef44e2b9b2f869ed3ed95e5, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.195557 21338 consensus_queue.cc:260] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.195657 21338 ts_tablet_manager.cc:1428] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.196348 21363 ts_tablet_manager.cc:1918] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:37.196410 21363 log.cc:1196] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/4aeca1e103fd4d7983e5a49190232bc3
I20260620 19:12:37.196619 21363 ts_tablet_manager.cc:1939] T 4aeca1e103fd4d7983e5a49190232bc3 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting consensus metadata
I20260620 19:12:37.196962 21362 tablet_replica.cc:330] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: stopping tablet replica
I20260620 19:12:37.196995 20821 catalog_manager.cc:4841] TS 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761): tablet 4aeca1e103fd4d7983e5a49190232bc3 (table test-table [id=8e0f4106c92c478593dd306c228c724c]) successfully deleted
I20260620 19:12:37.197082 21362 raft_consensus.cc:2229] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.197193 21362 raft_consensus.cc:2258] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.197350 21361 tablet_replica.cc:330] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: stopping tablet replica
I20260620 19:12:37.197453 21361 raft_consensus.cc:2229] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:37.197570 21361 raft_consensus.cc:2258] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.197619 21362 ts_tablet_manager.cc:1905] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.197983 21361 ts_tablet_manager.cc:1905] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.198722 21362 ts_tablet_manager.cc:1918] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:37.198782 21362 log.cc:1196] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/4aeca1e103fd4d7983e5a49190232bc3
I20260620 19:12:37.199038 21362 ts_tablet_manager.cc:1939] T 4aeca1e103fd4d7983e5a49190232bc3 P a53cde13e4b24fa8bb0d54e510014eca: Deleting consensus metadata
I20260620 19:12:37.199391 20820 catalog_manager.cc:4841] TS a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653): tablet 4aeca1e103fd4d7983e5a49190232bc3 (table test-table [id=8e0f4106c92c478593dd306c228c724c]) successfully deleted
I20260620 19:12:37.199496 21361 ts_tablet_manager.cc:1918] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:37.199564 21361 log.cc:1196] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/4aeca1e103fd4d7983e5a49190232bc3
I20260620 19:12:37.199810 21361 ts_tablet_manager.cc:1939] T 4aeca1e103fd4d7983e5a49190232bc3 P 0cabec436f2644bba9db53aca9c6750f: Deleting consensus metadata
I20260620 19:12:37.200150 20819 catalog_manager.cc:4841] TS 0cabec436f2644bba9db53aca9c6750f (127.19.66.193:34099): tablet 4aeca1e103fd4d7983e5a49190232bc3 (table test-table [id=8e0f4106c92c478593dd306c228c724c]) successfully deleted
I20260620 19:12:37.201246 21356 raft_consensus.cc:482] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:37.201308 21356 raft_consensus.cc:504] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.201445 21356 leader_election.cc:288] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761), a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653)
I20260620 19:12:37.201610 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248f4e8b95b4cf6ac44cbce0683aeb9" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53cde13e4b24fa8bb0d54e510014eca" is_pre_election: true
I20260620 19:12:37.201610 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248f4e8b95b4cf6ac44cbce0683aeb9" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" is_pre_election: true
I20260620 19:12:37.201683 21132 raft_consensus.cc:2454] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 0.
I20260620 19:12:37.201683 21272 raft_consensus.cc:2454] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 0.
I20260620 19:12:37.201821 20924 leader_election.cc:302] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cabec436f2644bba9db53aca9c6750f, a53cde13e4b24fa8bb0d54e510014eca; no voters:
I20260620 19:12:37.201970 21356 raft_consensus.cc:2781] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:37.202018 21356 raft_consensus.cc:482] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:37.202044 21356 raft_consensus.cc:3037] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.202710 21356 raft_consensus.cc:504] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.202833 21356 leader_election.cc:288] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 election: Requested vote from peers 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761), a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653)
I20260620 19:12:37.202941 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248f4e8b95b4cf6ac44cbce0683aeb9" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
I20260620 19:12:37.203002 21132 raft_consensus.cc:3037] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.202989 21272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0248f4e8b95b4cf6ac44cbce0683aeb9" candidate_uuid: "0cabec436f2644bba9db53aca9c6750f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53cde13e4b24fa8bb0d54e510014eca"
I20260620 19:12:37.203037 21272 raft_consensus.cc:3037] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.203613 21132 raft_consensus.cc:2454] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 1.
I20260620 19:12:37.203660 21272 raft_consensus.cc:2454] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cabec436f2644bba9db53aca9c6750f in term 1.
I20260620 19:12:37.203787 20925 leader_election.cc:302] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cabec436f2644bba9db53aca9c6750f, 479813a51ef44e2b9b2f869ed3ed95e5; no voters:
I20260620 19:12:37.203886 21356 raft_consensus.cc:2781] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:37.203930 21356 raft_consensus.cc:686] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 1 LEADER]: Becoming Leader. State: Replica: 0cabec436f2644bba9db53aca9c6750f, State: Running, Role: LEADER
I20260620 19:12:37.203975 21356 consensus_queue.cc:237] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.204464 20834 catalog_manager.cc:5495] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cabec436f2644bba9db53aca9c6750f (127.19.66.193). New cstate: current_term: 1 leader_uuid: "0cabec436f2644bba9db53aca9c6750f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982756, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:37.228519 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.221504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35979 (local address 127.0.0.1:35828)
0620 19:12:37.221630 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.221641 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.221698 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.221828 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.221829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.222120 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:37.222168 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.222170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.223076 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.223076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.223598 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.223600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.224037 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.224038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.224074 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.224576 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:37.224580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:37.226043 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:37.227634 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.227640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.227642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.227821 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.227994 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:37.227995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:37.227995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:37.228044 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:37.228269 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:37.228273 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:37.228309 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.228354 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.228446 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":67,"threads_started":1}
I20260620 19:12:37.228617 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.221565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.19.66.254:35979)
0620 19:12:37.221620 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.221622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.221697 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:12:37.221742 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.221742 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.221753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.221819 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.222230 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.223008 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.223662 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.223975 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.224042 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.226132 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.226134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.226134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.226142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.227566 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.227882 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.227883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.227883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.227943 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.228090 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.228090 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.228091 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.228215 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.228282 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.228382 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.228508 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":29}
I20260620 19:12:37.230403 20834 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:35828:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:12:37.230609 20834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:37.231912 21112 tablet_service.cc:1467] Processing CreateTablet for tablet fe126f989e94462e986c2465384691eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=242cb24ae74d4e81a1c277088b6881af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:37.231984 20972 tablet_service.cc:1467] Processing CreateTablet for tablet fe126f989e94462e986c2465384691eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=242cb24ae74d4e81a1c277088b6881af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:37.232074 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe126f989e94462e986c2465384691eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.232110 20972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe126f989e94462e986c2465384691eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.233613 21339 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: Bootstrap starting.
I20260620 19:12:37.233655 21252 tablet_service.cc:1467] Processing CreateTablet for tablet fe126f989e94462e986c2465384691eb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=242cb24ae74d4e81a1c277088b6881af]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:37.233783 21252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe126f989e94462e986c2465384691eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:37.234059 21339 tablet_bootstrap.cc:654] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.234668 21339 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: No bootstrap required, opened a new log
I20260620 19:12:37.234714 21339 ts_tablet_manager.cc:1397] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:37.234812 21342 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: Bootstrap starting.
I20260620 19:12:37.234827 21338 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: Bootstrap starting.
I20260620 19:12:37.234850 21339 raft_consensus.cc:348] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.234910 21339 raft_consensus.cc:374] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.234930 21339 raft_consensus.cc:729] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cabec436f2644bba9db53aca9c6750f, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.234968 21339 consensus_queue.cc:260] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.235045 21339 ts_tablet_manager.cc:1428] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.235143 21356 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.235198 21356 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.235214 21338 tablet_bootstrap.cc:654] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.235215 21342 tablet_bootstrap.cc:654] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:37.237303 21338 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: No bootstrap required, opened a new log
I20260620 19:12:37.237351 21338 ts_tablet_manager.cc:1397] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:12:37.237476 21338 raft_consensus.cc:348] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.237493 21342 tablet_bootstrap.cc:492] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: No bootstrap required, opened a new log
I20260620 19:12:37.237520 21338 raft_consensus.cc:374] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.237532 21338 raft_consensus.cc:729] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479813a51ef44e2b9b2f869ed3ed95e5, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.237533 21342 ts_tablet_manager.cc:1397] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 19:12:37.237564 21338 consensus_queue.cc:260] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.237638 21338 ts_tablet_manager.cc:1428] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.237658 21342 raft_consensus.cc:348] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.237715 21342 raft_consensus.cc:374] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:37.237733 21342 raft_consensus.cc:729] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53cde13e4b24fa8bb0d54e510014eca, State: Initialized, Role: FOLLOWER
I20260620 19:12:37.237696 21343 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.237761 21343 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.237777 21342 consensus_queue.cc:260] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.237865 21342 ts_tablet_manager.cc:1428] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:37.237910 21346 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.238006 21346 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: This TxnStatusTablet replica's current role is: FOLLOWER
W20260620 19:12:37.250489 21039 tablet.cc:2369] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:37.301996 21319 tablet.cc:2369] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:37.346287 21346 raft_consensus.cc:482] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:37.346385 21346 raft_consensus.cc:504] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.346728 21346 leader_election.cc:288] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cabec436f2644bba9db53aca9c6750f (127.19.66.193:34099), 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761)
I20260620 19:12:37.349397 21211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.346893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36761 (local address 127.19.66.195:46617)
0620 19:12:37.346946 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.346956 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.346999 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.347128 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.347129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.347130 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.347196 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.347197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.347803 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.347804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.348604 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.348607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.349157 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.349158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.349236 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.349239 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.349285 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.349323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:12:37.349397 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.346968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:46617 (local address 127.19.66.194:36761)
0620 19:12:37.347010 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.347011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.347021 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:12:37.347061 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.347062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.347070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.347111 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.347255 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.347730 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.348685 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.349092 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.349154 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.349163 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.349294 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.349326 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:12:37.349630 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe126f989e94462e986c2465384691eb" candidate_uuid: "a53cde13e4b24fa8bb0d54e510014eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" is_pre_election: true
I20260620 19:12:37.349716 21132 raft_consensus.cc:2454] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a53cde13e4b24fa8bb0d54e510014eca in term 0.
I20260620 19:12:37.349782 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.346991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:46627 (local address 127.19.66.193:34099)
0620 19:12:37.347037 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.347039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.347122 (+ 83us) server_negotiation.cc:407] Connection header received
0620 19:12:37.347183 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.347184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.347194 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.347240 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:37.347392 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.347845 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.349090 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.349435 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.349496 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.349504 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.349682 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.349723 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:12:37.349786 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.346900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:34099 (local address 127.19.66.195:46627)
0620 19:12:37.347061 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.347070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.347123 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.347247 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.347248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.347249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:37.347333 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.347334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.347911 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.347912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.349013 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.349016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.349553 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.349555 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.349618 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.349622 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.349668 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.349704 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":49,"thread_start_us":55,"threads_started":1}
I20260620 19:12:37.349901 21205 leader_election.cc:302] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479813a51ef44e2b9b2f869ed3ed95e5, a53cde13e4b24fa8bb0d54e510014eca; no voters:
I20260620 19:12:37.350103 21346 raft_consensus.cc:2781] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:37.350045 20992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe126f989e94462e986c2465384691eb" candidate_uuid: "a53cde13e4b24fa8bb0d54e510014eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cabec436f2644bba9db53aca9c6750f" is_pre_election: true
I20260620 19:12:37.350143 21346 raft_consensus.cc:482] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:37.350162 21346 raft_consensus.cc:3037] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.350168 20992 raft_consensus.cc:2454] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a53cde13e4b24fa8bb0d54e510014eca in term 0.
I20260620 19:12:37.350963 21346 raft_consensus.cc:504] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.351082 21346 leader_election.cc:288] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [CANDIDATE]: Term 1 election: Requested vote from peers 0cabec436f2644bba9db53aca9c6750f (127.19.66.193:34099), 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761)
I20260620 19:12:37.351225 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe126f989e94462e986c2465384691eb" candidate_uuid: "a53cde13e4b24fa8bb0d54e510014eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479813a51ef44e2b9b2f869ed3ed95e5"
I20260620 19:12:37.351303 21132 raft_consensus.cc:3037] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.351267 20992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe126f989e94462e986c2465384691eb" candidate_uuid: "a53cde13e4b24fa8bb0d54e510014eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cabec436f2644bba9db53aca9c6750f"
I20260620 19:12:37.351348 20992 raft_consensus.cc:3037] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:37.351845 21132 raft_consensus.cc:2454] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53cde13e4b24fa8bb0d54e510014eca in term 1.
I20260620 19:12:37.352002 21205 leader_election.cc:302] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479813a51ef44e2b9b2f869ed3ed95e5, a53cde13e4b24fa8bb0d54e510014eca; no voters:
I20260620 19:12:37.351992 20992 raft_consensus.cc:2454] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a53cde13e4b24fa8bb0d54e510014eca in term 1.
I20260620 19:12:37.352095 21346 raft_consensus.cc:2781] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:37.352257 21346 raft_consensus.cc:686] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [term 1 LEADER]: Becoming Leader. State: Replica: a53cde13e4b24fa8bb0d54e510014eca, State: Running, Role: LEADER
I20260620 19:12:37.352329 21346 consensus_queue.cc:237] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } }
I20260620 19:12:37.352649 21378 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: TxnStatusTablet state changed. Reason: New leader a53cde13e4b24fa8bb0d54e510014eca. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.352739 21378 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:37.352984 20834 catalog_manager.cc:5495] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca reported cstate change: term changed from 0 to 1, leader changed from <none> to a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195). New cstate: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:37.353240 21380 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:12:37.353348 21380 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:12:37.383706 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.381391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40653 (local address 127.0.0.1:51368)
0620 19:12:37.381472 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.381484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.381534 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.381687 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.381688 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.381689 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:37.381738 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.381741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.382349 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.382351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.382916 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.382918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.383226 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.383228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.383293 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.383312 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.383543 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.383552 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.383590 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.383638 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:12:37.383702 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.381460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.19.66.195:40653)
0620 19:12:37.381509 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.381510 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.381519 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:37.381602 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.381603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.381623 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.381677 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.381795 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.382275 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.382986 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.383165 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.383218 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.383388 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.383485 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.383545 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.383604 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.383625 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:12:37.384706 21132 raft_consensus.cc:1261] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Refusing update from remote peer a53cde13e4b24fa8bb0d54e510014eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.384809 20992 raft_consensus.cc:1261] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Refusing update from remote peer a53cde13e4b24fa8bb0d54e510014eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.384989 21378 consensus_queue.cc:1035] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.385093 21346 consensus_queue.cc:1035] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.385635 21343 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: TxnStatusTablet state changed. Reason: New leader a53cde13e4b24fa8bb0d54e510014eca. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.385723 21343 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.385792 21356 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: TxnStatusTablet state changed. Reason: New leader a53cde13e4b24fa8bb0d54e510014eca. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.385857 21356 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.386473 21343 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.386544 21343 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 479813a51ef44e2b9b2f869ed3ed95e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.387001 21378 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.387069 21378 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:37.387864 21356 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.387936 21356 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P 0cabec436f2644bba9db53aca9c6750f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:37.388096 21378 tablet_replica.cc:439] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a53cde13e4b24fa8bb0d54e510014eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cabec436f2644bba9db53aca9c6750f" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 34099 } } peers { permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 } } peers { permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 } } }
I20260620 19:12:37.388170 21378 tablet_replica.cc:442] T fe126f989e94462e986c2465384691eb P a53cde13e4b24fa8bb0d54e510014eca: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:37.392047 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.389584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:46487 (local address 127.19.66.195:40653)
0620 19:12:37.389618 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.389620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.389714 (+ 94us) server_negotiation.cc:407] Connection header received
0620 19:12:37.389748 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.389749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.389758 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.389821 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.389900 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.390602 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.391293 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.391588 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.391649 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.391848 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.391874 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.391916 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.391985 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.392002 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:37.392112 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.389517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40653 (local address 127.19.66.193:46487)
0620 19:12:37.389651 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.389661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.389712 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.389823 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.389824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.389825 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:37.389865 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.389866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.390663 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.390663 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.391228 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.391229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.391686 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.391687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.391749 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.391765 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.391930 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.391932 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.391981 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.392032 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260620 19:12:37.393888 21272 raft_consensus.cc:1261] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Refusing update from remote peer 0cabec436f2644bba9db53aca9c6750f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.394009 21132 raft_consensus.cc:1261] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Refusing update from remote peer 0cabec436f2644bba9db53aca9c6750f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:37.394070 21356 consensus_queue.cc:1035] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53cde13e4b24fa8bb0d54e510014eca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.394166 21356 consensus_queue.cc:1035] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [LEADER]: Connected to new peer: Peer: permanent_uuid: "479813a51ef44e2b9b2f869ed3ed95e5" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:37.401978 21324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.399577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:34099 (local address 127.19.66.195:36039)
0620 19:12:37.399630 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:37.399640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:37.399674 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:37.399787 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:37.399787 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:37.399788 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:37.399844 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.399845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.400454 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.400455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.401065 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:37.401067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.401511 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.401512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:37.401588 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.401604 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.401803 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.401805 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:37.401845 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 19:12:37.401893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 19:12:37.402158 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.399666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:36039 (local address 127.19.66.193:34099)
0620 19:12:37.399700 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.399702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.399709 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:37.399728 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.399728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.399737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.399776 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:37.399894 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.400372 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.401131 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.401394 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.401454 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.401709 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:37.401745 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:37.401793 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.402070 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.402094 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:12:37.410856 20834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.410964 20834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:37.411540 20834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3443165e33d4d82ac766a8f1271cf7b: Sending DeleteTablet for 3 replicas of tablet 0248f4e8b95b4cf6ac44cbce0683aeb9
I20260620 19:12:37.411814 21112 tablet_service.cc:1514] Processing DeleteTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:57446
I20260620 19:12:37.411831 20972 tablet_service.cc:1514] Processing DeleteTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55428
I20260620 19:12:37.411931 21363 tablet_replica.cc:330] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: stopping tablet replica
I20260620 19:12:37.411926 21252 tablet_service.cc:1514] Processing DeleteTablet for tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:51364
I20260620 19:12:37.411998 21363 raft_consensus.cc:2229] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.412012 21362 tablet_replica.cc:330] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: stopping tablet replica
I20260620 19:12:37.412030 21363 raft_consensus.cc:2258] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.412057 21362 raft_consensus.cc:2229] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:37.412091 21362 raft_consensus.cc:2258] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.412091 21361 tablet_replica.cc:330] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: stopping tablet replica
I20260620 19:12:37.412137 21361 raft_consensus.cc:2229] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:37.412191 21361 raft_consensus.cc:2258] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:37.412497 21361 ts_tablet_manager.cc:1905] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.412828 21362 ts_tablet_manager.cc:1905] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.413005 21363 ts_tablet_manager.cc:1905] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:37.414181 21362 ts_tablet_manager.cc:1918] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:37.414240 21362 log.cc:1196] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/0248f4e8b95b4cf6ac44cbce0683aeb9
I20260620 19:12:37.414428 21362 ts_tablet_manager.cc:1939] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P a53cde13e4b24fa8bb0d54e510014eca: Deleting consensus metadata
I20260620 19:12:37.414534 21363 ts_tablet_manager.cc:1918] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:37.414592 21363 log.cc:1196] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/0248f4e8b95b4cf6ac44cbce0683aeb9
I20260620 19:12:37.414709 20820 catalog_manager.cc:4841] TS a53cde13e4b24fa8bb0d54e510014eca (127.19.66.195:40653): tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (table test-table [id=1e8137f62177445794db9ba7c5ce0dd6]) successfully deleted
I20260620 19:12:37.414793 21363 ts_tablet_manager.cc:1939] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 479813a51ef44e2b9b2f869ed3ed95e5: Deleting consensus metadata
I20260620 19:12:37.415055 20821 catalog_manager.cc:4841] TS 479813a51ef44e2b9b2f869ed3ed95e5 (127.19.66.194:36761): tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (table test-table [id=1e8137f62177445794db9ba7c5ce0dd6]) successfully deleted
I20260620 19:12:37.416395 21361 ts_tablet_manager.cc:1918] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:37.416460 21361 log.cc:1196] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/0248f4e8b95b4cf6ac44cbce0683aeb9
I20260620 19:12:37.416648 21361 ts_tablet_manager.cc:1939] T 0248f4e8b95b4cf6ac44cbce0683aeb9 P 0cabec436f2644bba9db53aca9c6750f: Deleting consensus metadata
I20260620 19:12:37.416957 20819 catalog_manager.cc:4841] TS 0cabec436f2644bba9db53aca9c6750f (127.19.66.193:34099): tablet 0248f4e8b95b4cf6ac44cbce0683aeb9 (table test-table [id=1e8137f62177445794db9ba7c5ce0dd6]) successfully deleted
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[20790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982757, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:37.424666 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.412835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.19.66.193:34099)
0620 19:12:37.413976 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.413978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.417023 (+ 3045us) server_negotiation.cc:407] Connection header received
0620 19:12:37.417131 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.417132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.417145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.417190 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.417717 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.418207 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.418879 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.419076 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.419128 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.422119 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.422127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.422127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.422134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.423633 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.424003 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.424005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.424006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.424055 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.424226 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.424226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.424227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.424351 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.424460 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.424540 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.424601 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117}
W20260620 19:12:37.425035 21032 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55450
I20260620 19:12:37.432219 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:37.425966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.19.66.254:35979)
0620 19:12:37.426014 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:37.426016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:37.426108 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:12:37.426152 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:37.426153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:37.426164 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:37.426215 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:37.426737 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.427479 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.428161 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:37.428470 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:37.428546 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:37.429634 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:37.429636 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:37.429637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:37.429645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:37.431061 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.431358 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.431359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.431360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.431421 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:37.431579 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:37.431580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:37.431581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:37.431716 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:37.431868 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:37.432073 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 19:12:37.432142 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 19:12:37.432632 20834 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35838
I20260620 19:12:37.433224 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20907
I20260620 19:12:37.439272 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21047
I20260620 19:12:37.443573 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21187
I20260620 19:12:37.448102 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 20803
2026-06-20T19:12:37Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2050 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[21400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[21400](info): set up 2 sockets
Jun 20 19:12:37 dist-test-slave-nq8r krb5kdc[21400](info): commencing operation
krb5kdc: starting...
W20260620 19:12:38.392284 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:38.499828 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:32963
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:32963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:38.581533 21413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:38.581748 21413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:38.581769 21413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:38.583009 21413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:38.583055 21413 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:12:38.583067 21413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:38.583078 21413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:38.583089 21413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:38.583099 21413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:38.584614 21413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:32963
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:32963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:38.584841 21413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:38.585054 21413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:38.589591 21418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.589591 21421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.589601 21419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:38.589759 21413 server_base.cc:1034] running on GCE node
I20260620 19:12:38.590049 21413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:38.590345 21413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:38.591491 21413 hybrid_clock.cc:648] HybridClock initialized: now 1781982758591481 us; error 27 us; skew 500 ppm
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:38.593861 21413 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:38.594319 21413 webserver.cc:460] Webserver started at http://127.19.66.254:43253/ using document root <none> and password file <none>
I20260620 19:12:38.594514 21413 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:38.594554 21413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:38.594642 21413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:38.595623 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "4ef0ec1c279d4631bca48804f372a971"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "d7fffb185f6802b86e17e469d0764906"
server_key_iv: "49a6b6ab319e7545033d6fa296cca624"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.595929 21413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "4ef0ec1c279d4631bca48804f372a971"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "d7fffb185f6802b86e17e469d0764906"
server_key_iv: "49a6b6ab319e7545033d6fa296cca624"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.597136 21413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:38.597751 21428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.597947 21413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:38.598021 21413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "4ef0ec1c279d4631bca48804f372a971"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "d7fffb185f6802b86e17e469d0764906"
server_key_iv: "49a6b6ab319e7545033d6fa296cca624"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.598074 21413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:38.619237 21413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:38.621006 21413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:38.621117 21413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:38.625517 21413 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:32963
I20260620 19:12:38.625594 21490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:32963 every 8 connection(s)
I20260620 19:12:38.625885 21413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:38.626534 21491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:38.628729 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Bootstrap starting.
I20260620 19:12:38.629307 21491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:38.629549 21491 log.cc:824] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:38.630175 21491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: No bootstrap required, opened a new log
I20260620 19:12:38.632148 21491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } }
I20260620 19:12:38.632287 21491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:38.632318 21491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ef0ec1c279d4631bca48804f372a971, State: Initialized, Role: FOLLOWER
I20260620 19:12:38.632440 21491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } }
I20260620 19:12:38.632498 21491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:38.632527 21491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:38.632572 21491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:38.633080 21491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } }
I20260620 19:12:38.633169 21491 leader_election.cc:302] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ef0ec1c279d4631bca48804f372a971; no voters:
I20260620 19:12:38.633315 21491 leader_election.cc:288] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:38.633347 21494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:38.633486 21494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [term 1 LEADER]: Becoming Leader. State: Replica: 4ef0ec1c279d4631bca48804f372a971, State: Running, Role: LEADER
I20260620 19:12:38.633548 21491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:38.633575 21494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } }
I20260620 19:12:38.633922 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ef0ec1c279d4631bca48804f372a971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } } }
I20260620 19:12:38.633944 21496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ef0ec1c279d4631bca48804f372a971. Latest consensus state: current_term: 1 leader_uuid: "4ef0ec1c279d4631bca48804f372a971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ef0ec1c279d4631bca48804f372a971" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 32963 } } }
I20260620 19:12:38.634094 21496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:38.634043 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:38.634393 21503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:38.634606 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21413
I20260620 19:12:38.634685 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:38.634837 19723 external_mini_cluster.cc:1428] Setting key fdd5d13275422892443dce43fa5c632c
I20260620 19:12:38.634903 21503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:38.636343 21503 catalog_manager.cc:1347] Generated new cluster ID: 0a11d9abae194c5091f50c36bdba1458
I20260620 19:12:38.636394 21503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:38.643915 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.635408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.19.66.254:32963)
0620 19:12:38.635552 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:38.635561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:38.635592 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:12:38.635718 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:38.635725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:38.635801 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:38.635945 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:38.636688 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.637610 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.638320 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.638768 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.638864 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.641004 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:38.641008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:38.641008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:38.641020 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:38.642564 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.642869 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.642872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.642872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.642966 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.643136 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.643137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.643137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.643368 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:38.643514 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:38.643579 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:12:38.643707 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260620 19:12:38.656271 21503 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:38.656803 21503 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:38.660476 21503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Generated new TSK 0
I20260620 19:12:38.660621 21503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:38.689246 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:32963
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:38.777618 21517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:38.777818 21517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:38.777844 21517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:38.779000 21517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:38.779045 21517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:38.779064 21517 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:12:38.779085 21517 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:38.779107 21517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:38.780591 21517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:38.780825 21517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:38.781077 21517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:38.785725 21525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.785713 21523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.785727 21522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:38.785992 21517 server_base.cc:1034] running on GCE node
I20260620 19:12:38.786170 21517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:38.786412 21517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:38.787565 21517 hybrid_clock.cc:648] HybridClock initialized: now 1781982758787560 us; error 39 us; skew 500 ppm
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:38.790014 21517 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:38.790467 21517 webserver.cc:460] Webserver started at http://127.19.66.193:36685/ using document root <none> and password file <none>
I20260620 19:12:38.790674 21517 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:38.790720 21517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:38.790845 21517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:38.791858 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "d093f700c34f4d9a86a712d5e3971a86"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "306aa9739c1d2de8e3a8895a678af963"
server_key_iv: "bf76a5392b90092d278c76472317da4b"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.792183 21517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "d093f700c34f4d9a86a712d5e3971a86"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "306aa9739c1d2de8e3a8895a678af963"
server_key_iv: "bf76a5392b90092d278c76472317da4b"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.793489 21517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:38.794183 21532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.794384 21517 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:38.794485 21517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "d093f700c34f4d9a86a712d5e3971a86"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "306aa9739c1d2de8e3a8895a678af963"
server_key_iv: "bf76a5392b90092d278c76472317da4b"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.794561 21517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:38.812793 21517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:38.814646 21517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:38.814755 21517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:38.815244 21517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:38.815284 21517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.815312 21517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:38.815323 21517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:38.821939 21517 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:42855
I20260620 19:12:38.822006 21647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:42855 every 8 connection(s)
I20260620 19:12:38.822326 21517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:38.824172 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21517
I20260620 19:12:38.824273 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:38.824443 19723 external_mini_cluster.cc:1428] Setting key 1a408359b63707c2c982a3704da0d349
I20260620 19:12:38.824774 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.815645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:53149 (local address 127.19.66.254:32963)
0620 19:12:38.815697 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:38.815699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:38.815916 (+ 217us) server_negotiation.cc:407] Connection header received
0620 19:12:38.816406 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:38.816407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:38.816421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:38.816471 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:38.817174 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.817748 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.818753 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.818963 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.819026 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.822205 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:38.822206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:38.822206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:38.822213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:38.823455 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.823873 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.823874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.823874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.823920 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.824088 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.824089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.824090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.824223 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:38.824292 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:38.824603 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:12:38.824687 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:12:38.824991 21541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.815652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.193:53149)
0620 19:12:38.815801 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.815838 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:38.816308 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:38.816548 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:38.816552 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:38.816872 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:38.817076 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.817082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.817825 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.817827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.818663 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.818665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.819039 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.819040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.819175 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.819683 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:38.819693 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:38.822101 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:38.823587 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.823593 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.823595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.823808 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.823954 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.823955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.823955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.824038 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.824312 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:38.824319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:38.824464 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 19:12:38.824576 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:12:38.824741 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260620 19:12:38.826392 21648 heartbeater.cc:347] Connected to a master server at 127.19.66.254:32963
I20260620 19:12:38.826532 21648 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:38.826749 21648 heartbeater.cc:511] Master 127.19.66.254:32963 requested a full tablet report, sending...
I20260620 19:12:38.827221 21445 ts_manager.cc:194] Registered new tserver with Master: d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855)
I20260620 19:12:38.827813 21445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:53149
I20260620 19:12:38.831634 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.826272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.193:44177)
0620 19:12:38.826386 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.826396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:38.826449 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:38.826563 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:38.826563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:38.826717 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:38.826770 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.826772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.827494 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.827495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.828057 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.828059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.828460 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.828461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.828527 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.828938 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:38.828943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:38.829212 (+ 269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:38.830763 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.830764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.830764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.830922 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.831110 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.831110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.831111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.831159 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.831402 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:38.831402 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:38.831440 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:12:38.831497 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:12:38.831568 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260620 19:12:38.831696 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.826340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:44177 (local address 127.19.66.254:32963)
0620 19:12:38.826382 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:38.826383 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:38.826455 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:12:38.826498 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:38.826499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:38.826509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:38.826557 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:38.826854 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.827405 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.828129 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.828388 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.828452 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.829284 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:38.829286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:38.829286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:38.829292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:38.830678 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.830996 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.830997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.830997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.831050 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.831205 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.831206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.831206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.831325 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:38.831395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:38.831535 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 19:12:38.831609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:38.846751 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:32963
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:38.924793 21657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:38.925076 21657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:38.925114 21657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:38.926342 21657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:38.926384 21657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:38.926403 21657 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:12:38.926414 21657 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:38.926430 21657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:38.927901 21657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:38.928123 21657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:38.928326 21657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:38.932969 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.933013 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:38.932997 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:38.933164 21657 server_base.cc:1034] running on GCE node
I20260620 19:12:38.933341 21657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:38.933545 21657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:38.934670 21657 hybrid_clock.cc:648] HybridClock initialized: now 1781982758934663 us; error 29 us; skew 500 ppm
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:38.936971 21657 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:38.937335 21657 webserver.cc:460] Webserver started at http://127.19.66.194:41731/ using document root <none> and password file <none>
I20260620 19:12:38.937527 21657 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:38.937565 21657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:38.937655 21657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:38.938627 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "4f7a3f9842e44500b3d08ec805dfbefc"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "223ee6760477e2c692f6d58763c4a540"
server_key_iv: "30e9ea7aca8a9fa85befa2c3e49132d9"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.938903 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "4f7a3f9842e44500b3d08ec805dfbefc"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "223ee6760477e2c692f6d58763c4a540"
server_key_iv: "30e9ea7aca8a9fa85befa2c3e49132d9"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.940083 21657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:38.940711 21672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.940867 21657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:38.940946 21657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "4f7a3f9842e44500b3d08ec805dfbefc"
format_stamp: "Formatted at 2026-06-20 19:12:38 on dist-test-slave-nq8r"
server_key: "223ee6760477e2c692f6d58763c4a540"
server_key_iv: "30e9ea7aca8a9fa85befa2c3e49132d9"
server_key_version: "encryptionkey@0"
I20260620 19:12:38.941011 21657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:38.954428 21657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:38.956053 21657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:38.956156 21657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:38.956576 21657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:38.956614 21657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.956635 21657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:38.956646 21657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:38.962997 21657 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:42309
I20260620 19:12:38.963078 21787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:42309 every 8 connection(s)
Jun 20 19:12:38 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:38.963337 21657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:38.966286 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.957040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:47601 (local address 127.19.66.254:32963)
0620 19:12:38.957080 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:38.957082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:38.957301 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:12:38.957783 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:38.957785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:38.957798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:38.957856 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:38.958711 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.959467 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.960967 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.961264 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.961326 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.963817 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:38.963818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:38.963818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:38.963825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:38.965110 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.965455 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.965456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.965456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.965508 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.965714 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.965714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.965715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.965820 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:38.965874 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:38.966124 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:12:38.966199 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:12:38.966435 21681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.957048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.194:47601)
0620 19:12:38.957191 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.957224 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:38.957690 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:38.957968 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:38.957973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:38.958320 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:38.958610 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.958617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.959555 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.959555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.960874 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.960877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.961353 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.961354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.961505 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.962093 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:38.962104 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:38.963731 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:38.965178 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.965185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.965187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.965391 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.965560 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.965560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.965561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.965661 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.965871 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:38.965876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:38.966009 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 19:12:38.966095 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:12:38.966237 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260620 19:12:38.966868 21788 heartbeater.cc:347] Connected to a master server at 127.19.66.254:32963
I20260620 19:12:38.966948 21788 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:38.967118 21788 heartbeater.cc:511] Master 127.19.66.254:32963 requested a full tablet report, sending...
I20260620 19:12:38.967446 21444 ts_manager.cc:194] Registered new tserver with Master: 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309)
I20260620 19:12:38.967854 21444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:47601
I20260620 19:12:38.970901 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21657
I20260620 19:12:38.970999 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:38.971165 19723 external_mini_cluster.cc:1428] Setting key 0814cc5c2e5dc8ecb8dcffad49ee8f6a
I20260620 19:12:38.972604 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.967863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:53571 (local address 127.19.66.254:32963)
0620 19:12:38.967902 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:38.967903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:38.967957 (+ 54us) server_negotiation.cc:407] Connection header received
0620 19:12:38.968029 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:38.968030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:38.968039 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:38.968086 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:38.968345 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.968788 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.969451 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.969642 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.969693 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.970466 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:38.970467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:38.970467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:38.970473 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:38.971651 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.971930 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.971931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.971932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.971982 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:38.972134 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:38.972134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:38.972135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:38.972265 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:38.972320 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:38.972458 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:12:38.972520 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:38.972620 21794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:38.967791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.194:53571)
0620 19:12:38.967900 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:38.967912 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:38.967970 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:38.968088 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:38.968089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:38.968230 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:38.968278 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.968279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.968887 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.968888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.969378 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:38.969379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:38.969696 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:38.969697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:38.969771 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:38.970137 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:38.970141 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:38.970383 (+ 242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:38.971720 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.971721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.971721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.971873 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.972032 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:38.972032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:38.972033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:38.972086 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:38.972320 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:38.972320 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:38.972364 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 19:12:38.972431 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:12:38.972522 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:38.999665 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:32963
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:41131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:39.076421 21797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:39.076627 21797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:39.076649 21797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:39.077797 21797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:39.077840 21797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:39.077859 21797 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 19:12:39.077870 21797 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:12:39.077886 21797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:39.079267 21797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:39.079476 21797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:39.079690 21797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:39.084216 21803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:39.084224 21802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:39.084240 21805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:39.084443 21797 server_base.cc:1034] running on GCE node
I20260620 19:12:39.084676 21797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:39.084923 21797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:39.086059 21797 hybrid_clock.cc:648] HybridClock initialized: now 1781982759086046 us; error 29 us; skew 500 ppm
Jun 20 19:12:39 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982759, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:39.088375 21797 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:39.088793 21797 webserver.cc:460] Webserver started at http://127.19.66.195:32925/ using document root <none> and password file <none>
I20260620 19:12:39.089007 21797 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:39.089053 21797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:39.089170 21797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:39.090139 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "89c3aba1ab6446648945040d0c70282e"
format_stamp: "Formatted at 2026-06-20 19:12:39 on dist-test-slave-nq8r"
server_key: "0d3b59e05b1cfe4f4e7fdeae15f2eca7"
server_key_iv: "5509b8858c62b13007047fd7f7364e4c"
server_key_version: "encryptionkey@0"
I20260620 19:12:39.090451 21797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "89c3aba1ab6446648945040d0c70282e"
format_stamp: "Formatted at 2026-06-20 19:12:39 on dist-test-slave-nq8r"
server_key: "0d3b59e05b1cfe4f4e7fdeae15f2eca7"
server_key_iv: "5509b8858c62b13007047fd7f7364e4c"
server_key_version: "encryptionkey@0"
I20260620 19:12:39.091621 21797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:39.092270 21812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.092437 21797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:39.092520 21797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "89c3aba1ab6446648945040d0c70282e"
format_stamp: "Formatted at 2026-06-20 19:12:39 on dist-test-slave-nq8r"
server_key: "0d3b59e05b1cfe4f4e7fdeae15f2eca7"
server_key_iv: "5509b8858c62b13007047fd7f7364e4c"
server_key_version: "encryptionkey@0"
I20260620 19:12:39.092577 21797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:39.113581 21797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:39.115759 21797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:39.115914 21797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:39.116436 21797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:39.116477 21797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.116515 21797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:39.116539 21797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:12:39 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982759, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:39.123111 21797 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:36697
I20260620 19:12:39.123178 21927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:36697 every 8 connection(s)
I20260620 19:12:39.123450 21797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:39.124028 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 21797
I20260620 19:12:39.124104 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:39.124269 19723 external_mini_cluster.cc:1428] Setting key 271173ca7136d4656455f4843fd8c68d
I20260620 19:12:39.125399 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.116840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:60631 (local address 127.19.66.254:32963)
0620 19:12:39.116927 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.116929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.117148 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:12:39.117680 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.117682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.117696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.117747 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.118436 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.119021 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.119999 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.120196 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.120253 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.122590 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.122591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.122592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.122601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.124123 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.124521 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.124523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.124523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.124588 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.124806 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.124807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.124808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.124969 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.125034 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.125274 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.125337 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:12:39.125556 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.116869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.195:60631)
0620 19:12:39.117023 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.117060 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.117575 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.117819 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.117823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.118143 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:39.118342 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.118347 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.119107 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.119108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.119914 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.119916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.120264 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.120265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.120411 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.120930 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:39.120939 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:39.122502 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:39.124198 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.124203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.124205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.124443 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.124649 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.124649 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.124650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.124749 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.125032 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:39.125037 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:39.125164 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.125246 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.125362 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260620 19:12:39.126010 21928 heartbeater.cc:347] Connected to a master server at 127.19.66.254:32963
I20260620 19:12:39.126086 21928 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:39.126245 21928 heartbeater.cc:511] Master 127.19.66.254:32963 requested a full tablet report, sending...
I20260620 19:12:39.126600 21445 ts_manager.cc:194] Registered new tserver with Master: 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.127060 21445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:60631
I20260620 19:12:39.131678 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.126996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:48245 (local address 127.19.66.254:32963)
0620 19:12:39.127034 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.127036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.127145 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:12:39.127228 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.127229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.127238 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.127265 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.127610 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.128126 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.128758 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.128975 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.129022 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.129825 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.129826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.129826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.129830 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.130937 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.131212 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.131213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.131213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.131253 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.131359 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.131359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.131359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.131437 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.131477 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.131570 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.131624 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:12:39.131687 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.126930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.19.66.195:48245)
0620 19:12:39.127085 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.127095 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.127167 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.127316 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.127316 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.127486 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:39.127539 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.127541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.128195 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.128195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.128704 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.128705 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.129027 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.129027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.129081 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.129451 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:39.129456 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:39.129758 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:39.131005 (+ 1247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.131006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.131006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.131158 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.131287 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.131287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.131288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.131328 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.131479 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:39.131480 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:39.131511 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.131554 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.131621 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260620 19:12:39.135725 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:12:39 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:12:39 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:39.155625 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.149626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.19.66.254:32963)
0620 19:12:39.149683 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.149685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.149692 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:39.149716 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.149716 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.149725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.149778 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.150268 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.150757 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.151457 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.151633 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.151677 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.153643 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.153644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.153644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.153650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.154833 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.155116 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.155116 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.155116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.155157 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.155280 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.155280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.155280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.155368 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.155433 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.155511 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.155563 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:12:39.157135 21445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:39.158576 21445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:39.164294 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:42855 (local address 127.0.0.1:39276)
0620 19:12:39.160691 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.160732 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.160802 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.161050 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.161052 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.161054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:39.161121 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.161122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.162065 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.162065 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.163216 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.163219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.164012 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.164087 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164109 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.164172 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.164223 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":53,"threads_started":1}
I20260620 19:12:39.164310 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.19.66.193:42855)
0620 19:12:39.160686 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.160690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.160887 (+ 197us) server_negotiation.cc:407] Connection header received
0620 19:12:39.160933 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.160936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.160952 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.161048 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:39.161203 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.161999 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.163290 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.163942 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164007 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164064 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.164190 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.164227 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260620 19:12:39.164706 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42309 (local address 127.0.0.1:40730)
0620 19:12:39.160756 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.160766 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.160817 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.161094 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.161095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.161097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:39.161124 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.161125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.162508 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.162509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.163567 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.163569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.164494 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.164555 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164559 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.164604 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.164641 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":63,"threads_started":1}
I20260620 19:12:39.164795 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:36697 (local address 127.0.0.1:34124)
0620 19:12:39.161166 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.161175 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.161226 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.161432 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.161433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.161435 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:39.161485 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.161487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.162635 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.162636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.163705 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.163708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.164565 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.164640 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164644 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.164697 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.164733 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":55,"threads_started":1}
I20260620 19:12:39.165010 21946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.19.66.194:42309)
0620 19:12:39.160955 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.160960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.160968 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:39.161002 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.161004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.161017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.161087 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:39.161340 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.162414 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.163635 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.164339 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164406 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164458 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.164876 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.164926 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":90,"threads_started":1}
I20260620 19:12:39.165241 21582 tablet_service.cc:1467] Processing CreateTablet for tablet b20842e6f1684183ac954352770296c0 (DEFAULT_TABLE table=test-table [id=e4e6bf88211e41fea1aa70269c203ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.165620 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20842e6f1684183ac954352770296c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.165567 21722 tablet_service.cc:1467] Processing CreateTablet for tablet b20842e6f1684183ac954352770296c0 (DEFAULT_TABLE table=test-table [id=e4e6bf88211e41fea1aa70269c203ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.165849 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20842e6f1684183ac954352770296c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.168313 21948 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Bootstrap starting.
I20260620 19:12:39.168627 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.160795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.19.66.195:36697)
0620 19:12:39.160974 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.160978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.161223 (+ 245us) server_negotiation.cc:407] Connection header received
0620 19:12:39.161285 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.161287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.161302 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.161373 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:39.161561 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.162570 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.163792 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.164496 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.164556 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.164601 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.164722 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.164758 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260620 19:12:39.168988 21948 tablet_bootstrap.cc:654] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.169006 21949 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Bootstrap starting.
I20260620 19:12:39.169257 21948 log.cc:824] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:39.169214 21862 tablet_service.cc:1467] Processing CreateTablet for tablet b20842e6f1684183ac954352770296c0 (DEFAULT_TABLE table=test-table [id=e4e6bf88211e41fea1aa70269c203ebb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.169530 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b20842e6f1684183ac954352770296c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.169592 21949 tablet_bootstrap.cc:654] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.169855 21948 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: No bootstrap required, opened a new log
I20260620 19:12:39.169881 21949 log.cc:824] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:39.169921 21948 ts_tablet_manager.cc:1397] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:12:39.170570 21949 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: No bootstrap required, opened a new log
I20260620 19:12:39.170639 21949 ts_tablet_manager.cc:1397] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:39.171353 21948 raft_consensus.cc:348] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.171479 21948 raft_consensus.cc:374] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.171510 21948 raft_consensus.cc:729] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d093f700c34f4d9a86a712d5e3971a86, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.171618 21948 consensus_queue.cc:260] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.171859 21948 ts_tablet_manager.cc:1428] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:12:39.171867 21648 heartbeater.cc:503] Master 127.19.66.254:32963 was elected leader, sending a full tablet report...
I20260620 19:12:39.172454 21953 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Bootstrap starting.
I20260620 19:12:39.172760 21949 raft_consensus.cc:348] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.172932 21949 raft_consensus.cc:374] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.172971 21949 raft_consensus.cc:729] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.173082 21949 consensus_queue.cc:260] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.173271 21953 tablet_bootstrap.cc:654] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.173308 21949 ts_tablet_manager.cc:1428] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:39.173347 21788 heartbeater.cc:503] Master 127.19.66.254:32963 was elected leader, sending a full tablet report...
I20260620 19:12:39.173591 21953 log.cc:824] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:39.174320 21953 tablet_bootstrap.cc:492] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: No bootstrap required, opened a new log
I20260620 19:12:39.174368 21953 ts_tablet_manager.cc:1397] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:12:39.175603 21953 raft_consensus.cc:348] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.175724 21953 raft_consensus.cc:374] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.175765 21953 raft_consensus.cc:729] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.175880 21953 consensus_queue.cc:260] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.176100 21953 ts_tablet_manager.cc:1428] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:12:39.176149 21928 heartbeater.cc:503] Master 127.19.66.254:32963 was elected leader, sending a full tablet report...
W20260620 19:12:39.213891 21789 tablet.cc:2369] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:39.323271 21649 tablet.cc:2369] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:39.373973 21929 tablet.cc:2369] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:39.505031 21954 raft_consensus.cc:482] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:39.505175 21954 raft_consensus.cc:504] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.505529 21954 leader_election.cc:288] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.508401 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.506016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:43029 (local address 127.19.66.195:36697)
0620 19:12:39.506071 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.506073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.506108 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:39.506183 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.506184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.506192 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.506245 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:39.506394 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.506899 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.507745 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.508102 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.508166 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.508173 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.508298 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.508334 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:12:39.508404 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.505793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50481 (local address 127.19.66.193:42855)
0620 19:12:39.505891 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.505893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.505942 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:12:39.506028 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.506029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.506040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.506099 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:39.506263 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.506772 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.507732 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.508090 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.508146 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.508155 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.508302 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.508335 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:12:39.508446 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.505719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:42855 (local address 127.19.66.194:50481)
0620 19:12:39.505841 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.505853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.505962 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.506118 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.506119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.506123 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:39.506196 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.506198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.506850 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.506851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.507643 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.507645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.508166 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.508167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.508235 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.508239 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.508286 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.508327 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260620 19:12:39.508632 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.505880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:36697 (local address 127.19.66.194:43029)
0620 19:12:39.506062 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.506071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.506113 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.506258 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.506259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.506261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:39.506321 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.506322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.506976 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.506977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.507664 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.507666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.508178 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.508179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.508237 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.508240 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.508285 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.508326 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":62,"threads_started":1}
I20260620 19:12:39.508754 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20842e6f1684183ac954352770296c0" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86" is_pre_election: true
I20260620 19:12:39.508948 21602 raft_consensus.cc:2454] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.509050 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20842e6f1684183ac954352770296c0" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e" is_pre_election: true
I20260620 19:12:39.509189 21882 raft_consensus.cc:2454] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.509385 21675 leader_election.cc:302] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.509513 21954 raft_consensus.cc:2781] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:39.509567 21954 raft_consensus.cc:482] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:39.509596 21954 raft_consensus.cc:3037] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.510231 21954 raft_consensus.cc:504] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.510345 21954 leader_election.cc:288] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Requested vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.510527 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20842e6f1684183ac954352770296c0" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e"
I20260620 19:12:39.510535 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b20842e6f1684183ac954352770296c0" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86"
I20260620 19:12:39.510607 21882 raft_consensus.cc:3037] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.510609 21602 raft_consensus.cc:3037] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.511303 21882 raft_consensus.cc:2454] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.511384 21602 raft_consensus.cc:2454] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.511459 21675 leader_election.cc:302] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.511569 21954 raft_consensus.cc:2781] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:39.511718 21954 raft_consensus.cc:686] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Becoming Leader. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Running, Role: LEADER
I20260620 19:12:39.511808 21954 consensus_queue.cc:237] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } }
I20260620 19:12:39.512547 21445 catalog_manager.cc:5495] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194). New cstate: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } health_report { overall_health: HEALTHY } } }
Jun 20 19:12:39 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982758, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:39.619653 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.613493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:32963 (local address 127.0.0.1:41264)
0620 19:12:39.613676 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.613686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.613748 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.613859 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.613859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.614154 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:39.614203 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.614205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.614818 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.614819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.615382 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.615384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.615668 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.615668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.615705 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.616212 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:39.616216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:39.617580 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:39.618817 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.618824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.618826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.619036 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.619172 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:39.619173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:39.619173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:39.619223 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:39.619383 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:39.619387 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:39.619426 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.619470 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.619552 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260620 19:12:39.619664 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.613539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.19.66.254:32963)
0620 19:12:39.613600 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.613602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.613743 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:12:39.613799 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.613800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.613809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.613846 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:39.614262 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.614739 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.615445 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.615622 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.615663 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.617656 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:39.617658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:39.617658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:39.617664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:39.618741 (+ 1077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.619088 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.619089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.619089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.619130 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:39.619260 (+ 130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:39.619260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:39.619261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:39.619344 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:39.619386 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.619490 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.619591 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":33}
I20260620 19:12:39.621506 21445 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:41264:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:12:39.621677 21445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:39.622926 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3496beb4e24d00a25d1cb4a8ce083d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c706cce175e1468086cbdebcd9e4d245]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:39.622937 21582 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3496beb4e24d00a25d1cb4a8ce083d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c706cce175e1468086cbdebcd9e4d245]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:39.622927 21862 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3496beb4e24d00a25d1cb4a8ce083d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c706cce175e1468086cbdebcd9e4d245]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:12:39.623081 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3496beb4e24d00a25d1cb4a8ce083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.623081 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3496beb4e24d00a25d1cb4a8ce083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.623101 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f3496beb4e24d00a25d1cb4a8ce083d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.624140 21949 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: Bootstrap starting.
I20260620 19:12:39.624488 21953 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: Bootstrap starting.
I20260620 19:12:39.624500 21948 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: Bootstrap starting.
I20260620 19:12:39.624526 21949 tablet_bootstrap.cc:654] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.624790 21948 tablet_bootstrap.cc:654] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.625039 21953 tablet_bootstrap.cc:654] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.625531 21948 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: No bootstrap required, opened a new log
I20260620 19:12:39.625576 21948 ts_tablet_manager.cc:1397] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:39.625720 21948 raft_consensus.cc:348] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.625805 21948 raft_consensus.cc:374] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.625823 21948 raft_consensus.cc:729] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d093f700c34f4d9a86a712d5e3971a86, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.625866 21948 consensus_queue.cc:260] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.625964 21948 ts_tablet_manager.cc:1428] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.626034 21952 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.626109 21952 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.626349 21953 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: No bootstrap required, opened a new log
I20260620 19:12:39.626398 21953 ts_tablet_manager.cc:1397] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:39.626535 21953 raft_consensus.cc:348] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.626602 21953 raft_consensus.cc:374] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.626627 21953 raft_consensus.cc:729] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.626673 21953 consensus_queue.cc:260] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.626755 21953 ts_tablet_manager.cc:1428] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.626808 21956 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.626899 21956 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.627876 21949 tablet_bootstrap.cc:492] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: No bootstrap required, opened a new log
I20260620 19:12:39.627928 21949 ts_tablet_manager.cc:1397] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:12:39.628232 21949 raft_consensus.cc:348] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.628300 21949 raft_consensus.cc:374] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.628314 21949 raft_consensus.cc:729] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.628356 21949 consensus_queue.cc:260] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.628441 21949 ts_tablet_manager.cc:1428] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.628500 21954 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.628582 21954 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.657729 21954 raft_consensus.cc:482] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:39.657816 21954 raft_consensus.cc:504] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.657959 21954 leader_election.cc:288] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.658164 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3496beb4e24d00a25d1cb4a8ce083d" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86" is_pre_election: true
I20260620 19:12:39.658164 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3496beb4e24d00a25d1cb4a8ce083d" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e" is_pre_election: true
I20260620 19:12:39.658253 21882 raft_consensus.cc:2454] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.658257 21602 raft_consensus.cc:2454] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.658381 21675 leader_election.cc:302] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.658470 21954 raft_consensus.cc:2781] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:39.658507 21954 raft_consensus.cc:482] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:39.658531 21954 raft_consensus.cc:3037] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.659231 21954 raft_consensus.cc:504] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.659354 21954 leader_election.cc:288] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Requested vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.659498 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3496beb4e24d00a25d1cb4a8ce083d" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e"
I20260620 19:12:39.659492 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3496beb4e24d00a25d1cb4a8ce083d" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86"
I20260620 19:12:39.659562 21882 raft_consensus.cc:3037] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.659571 21602 raft_consensus.cc:3037] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.660099 21882 raft_consensus.cc:2454] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.660177 21602 raft_consensus.cc:2454] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.660257 21675 leader_election.cc:302] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.660347 21954 raft_consensus.cc:2781] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:39.660408 21954 raft_consensus.cc:686] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Becoming Leader. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Running, Role: LEADER
I20260620 19:12:39.660459 21954 consensus_queue.cc:237] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.660754 21959 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: TxnStatusTablet state changed. Reason: New leader 4f7a3f9842e44500b3d08ec805dfbefc. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.660828 21959 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:39.661032 21971 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:12:39.661149 21971 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:12:39.661072 21445 catalog_manager.cc:5495] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194). New cstate: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } health_report { overall_health: UNKNOWN } } }
I20260620 19:12:39.673091 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.670697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.19.66.194:42309)
0620 19:12:39.670803 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.670805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.670812 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:39.670844 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.670844 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.670862 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.670922 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:39.671057 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.671546 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.672322 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.672538 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.672589 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.672749 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:39.672876 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:39.672954 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.672972 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.672991 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":42,"threads_started":1}
I20260620 19:12:39.673084 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.670642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42309 (local address 127.0.0.1:40744)
0620 19:12:39.670702 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.670714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.670753 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.670932 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.670933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.670934 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:39.670983 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.670986 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.671610 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.671612 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.672247 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.672249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.672593 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.672594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.672658 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.672675 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:39.672924 (+ 249us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:39.672932 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.672967 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.673013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:12:39.674623 21882 raft_consensus.cc:1261] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.674664 21602 raft_consensus.cc:1261] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.674976 21954 consensus_queue.cc:1035] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:39.675114 21959 consensus_queue.cc:1035] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:39.676177 21956 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: TxnStatusTablet state changed. Reason: New leader 4f7a3f9842e44500b3d08ec805dfbefc. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.676258 21956 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.676249 21952 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: TxnStatusTablet state changed. Reason: New leader 4f7a3f9842e44500b3d08ec805dfbefc. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.676328 21952 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.677417 21954 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.677495 21954 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:39.677534 21954 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.677569 21954 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 4f7a3f9842e44500b3d08ec805dfbefc: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:12:39.677588 21952 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.677665 21952 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P d093f700c34f4d9a86a712d5e3971a86: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.677680 21977 mvcc.cc:205] Tried to move back new op lower bound from 7299001383625125888 to 7299001383569633280. Current Snapshot: MvccSnapshot[applied={T|T < 7299001383625125888}]
I20260620 19:12:39.678145 21956 tablet_replica.cc:439] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } } }
I20260620 19:12:39.678220 21956 tablet_replica.cc:442] T 2f3496beb4e24d00a25d1cb4a8ce083d P 89c3aba1ab6446648945040d0c70282e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:12:39.679111 21976 mvcc.cc:205] Tried to move back new op lower bound from 7299001383625125888 to 7299001383569633280. Current Snapshot: MvccSnapshot[applied={T|T < 7299001383625125888}]
I20260620 19:12:39.682096 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.679559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.19.66.194:42309)
0620 19:12:39.679599 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.679601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.679736 (+ 135us) server_negotiation.cc:407] Connection header received
0620 19:12:39.679811 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.679812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.679822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.679850 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:39.680012 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.680459 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.681564 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.681735 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.681784 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.681923 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:39.681946 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:39.681979 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.682035 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.682050 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:12:39.686230 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.684109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:57525 (local address 127.19.66.194:42309)
0620 19:12:39.684136 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:39.684138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:39.684252 (+ 114us) server_negotiation.cc:407] Connection header received
0620 19:12:39.684304 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:39.684304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:39.684312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:39.684349 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:39.684468 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.684942 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.685643 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.685828 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.685876 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.686026 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:39.686053 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:39.686095 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:39.686140 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:12:39.686158 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12}
I20260620 19:12:39.686342 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:39.684072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42309 (local address 127.19.66.194:57525)
0620 19:12:39.684197 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:39.684205 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:39.684254 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:39.684361 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:39.684361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:39.684363 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:39.684415 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.684416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.684999 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.685000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.685579 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:39.685581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:39.685878 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:39.685879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:39.685934 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:39.685955 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:39.686100 (+ 145us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:39.686101 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:12:39.686139 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:12:39.686174 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260620 19:12:39.688055 21882 raft_consensus.cc:1261] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.688102 21602 raft_consensus.cc:1261] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.688225 21959 consensus_queue.cc:1035] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:39.688324 21954 consensus_queue.cc:1035] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:39.689522 21975 mvcc.cc:205] Tried to move back new op lower bound from 7299001383681257472 to 7299001382960947200. Current Snapshot: MvccSnapshot[applied={T|T < 7299001383681257472}]
I20260620 19:12:39.699209 21444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:39.699316 21444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:39.699959 21444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Sending DeleteTablet for 3 replicas of tablet b20842e6f1684183ac954352770296c0
I20260620 19:12:39.700224 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet b20842e6f1684183ac954352770296c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:39 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:40730
I20260620 19:12:39.700238 21862 tablet_service.cc:1514] Processing DeleteTablet for tablet b20842e6f1684183ac954352770296c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:39 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260620 19:12:39.700459 21992 tablet_replica.cc:330] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: stopping tablet replica
I20260620 19:12:39.700515 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet b20842e6f1684183ac954352770296c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:39 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39276
I20260620 19:12:39.700553 21992 raft_consensus.cc:2229] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:39.700635 21992 raft_consensus.cc:2258] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:39.700695 21994 tablet_replica.cc:330] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: stopping tablet replica
I20260620 19:12:39.700929 21994 raft_consensus.cc:2229] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:39.700976 21994 raft_consensus.cc:2258] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:39.700970 21992 ts_tablet_manager.cc:1905] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:39.700969 21444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:39.701094 21444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:39.701350 21994 ts_tablet_manager.cc:1905] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:39.702327 21582 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 (DEFAULT_TABLE table=test-table [id=86f276d9c41b43fd9b8118d0c9601020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.702430 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5448e5ec544b67a68965e0262b5a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.702632 21992 ts_tablet_manager.cc:1918] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:39.702684 21992 log.cc:1196] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/b20842e6f1684183ac954352770296c0
I20260620 19:12:39.702907 21992 ts_tablet_manager.cc:1939] T b20842e6f1684183ac954352770296c0 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting consensus metadata
I20260620 19:12:39.703248 21994 ts_tablet_manager.cc:1918] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:39.703305 21994 log.cc:1196] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/b20842e6f1684183ac954352770296c0
I20260620 19:12:39.703526 21994 ts_tablet_manager.cc:1939] T b20842e6f1684183ac954352770296c0 P d093f700c34f4d9a86a712d5e3971a86: Deleting consensus metadata
I20260620 19:12:39.703593 21948 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Bootstrap starting.
I20260620 19:12:39.703691 21429 catalog_manager.cc:4841] TS 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309): tablet b20842e6f1684183ac954352770296c0 (table test-table [id=e4e6bf88211e41fea1aa70269c203ebb]) successfully deleted
I20260620 19:12:39.703774 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 (DEFAULT_TABLE table=test-table [id=86f276d9c41b43fd9b8118d0c9601020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.703858 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5448e5ec544b67a68965e0262b5a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.703958 21948 tablet_bootstrap.cc:654] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.704393 21862 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 (DEFAULT_TABLE table=test-table [id=86f276d9c41b43fd9b8118d0c9601020]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:39.704504 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5448e5ec544b67a68965e0262b5a49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:39.704969 21949 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Bootstrap starting.
I20260620 19:12:39.705333 21949 tablet_bootstrap.cc:654] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.705338 21431 catalog_manager.cc:4841] TS d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855): tablet b20842e6f1684183ac954352770296c0 (table test-table [id=e4e6bf88211e41fea1aa70269c203ebb]) successfully deleted
I20260620 19:12:39.705627 21953 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Bootstrap starting.
I20260620 19:12:39.705879 21993 tablet_replica.cc:330] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: stopping tablet replica
I20260620 19:12:39.705992 21993 raft_consensus.cc:2229] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:39.706018 21953 tablet_bootstrap.cc:654] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:39.706058 21993 raft_consensus.cc:2258] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:39.706357 21993 ts_tablet_manager.cc:1905] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:39.706583 21949 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: No bootstrap required, opened a new log
I20260620 19:12:39.706632 21949 ts_tablet_manager.cc:1397] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:39.706763 21949 raft_consensus.cc:348] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.706830 21949 raft_consensus.cc:374] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.706848 21949 raft_consensus.cc:729] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.706890 21949 consensus_queue.cc:260] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.706979 21949 ts_tablet_manager.cc:1428] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.707181 21953 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: No bootstrap required, opened a new log
I20260620 19:12:39.707224 21953 ts_tablet_manager.cc:1397] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:39.707350 21953 raft_consensus.cc:348] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.707419 21953 raft_consensus.cc:374] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.707438 21953 raft_consensus.cc:729] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.707474 21953 consensus_queue.cc:260] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.707568 21953 ts_tablet_manager.cc:1428] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.707983 21993 ts_tablet_manager.cc:1918] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:39.708051 21993 log.cc:1196] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/b20842e6f1684183ac954352770296c0
I20260620 19:12:39.708256 21993 ts_tablet_manager.cc:1939] T b20842e6f1684183ac954352770296c0 P 89c3aba1ab6446648945040d0c70282e: Deleting consensus metadata
I20260620 19:12:39.708590 21431 catalog_manager.cc:4841] TS 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697): tablet b20842e6f1684183ac954352770296c0 (table test-table [id=e4e6bf88211e41fea1aa70269c203ebb]) successfully deleted
I20260620 19:12:39.709187 21948 tablet_bootstrap.cc:492] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: No bootstrap required, opened a new log
I20260620 19:12:39.709239 21948 ts_tablet_manager.cc:1397] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 19:12:39.709364 21948 raft_consensus.cc:348] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.709424 21948 raft_consensus.cc:374] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:39.709441 21948 raft_consensus.cc:729] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d093f700c34f4d9a86a712d5e3971a86, State: Initialized, Role: FOLLOWER
I20260620 19:12:39.709482 21948 consensus_queue.cc:260] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.709564 21948 ts_tablet_manager.cc:1428] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:39.718866 21959 raft_consensus.cc:482] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:39.718940 21959 raft_consensus.cc:504] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.719064 21959 leader_election.cc:288] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.719216 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5448e5ec544b67a68965e0262b5a49" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86" is_pre_election: true
I20260620 19:12:39.719233 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5448e5ec544b67a68965e0262b5a49" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e" is_pre_election: true
I20260620 19:12:39.719301 21882 raft_consensus.cc:2454] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.719305 21602 raft_consensus.cc:2454] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:39.719439 21675 leader_election.cc:302] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.719544 21959 raft_consensus.cc:2781] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:39.719586 21959 raft_consensus.cc:482] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:39.719605 21959 raft_consensus.cc:3037] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.720129 21959 raft_consensus.cc:504] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.720235 21959 leader_election.cc:288] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Requested vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:39.720368 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5448e5ec544b67a68965e0262b5a49" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86"
I20260620 19:12:39.720438 21602 raft_consensus.cc:3037] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.720420 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5448e5ec544b67a68965e0262b5a49" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e"
I20260620 19:12:39.720471 21882 raft_consensus.cc:3037] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:39.721113 21882 raft_consensus.cc:2454] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.721127 21602 raft_consensus.cc:2454] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:39.721256 21675 leader_election.cc:302] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:39.721354 21959 raft_consensus.cc:2781] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:39.721402 21959 raft_consensus.cc:686] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Becoming Leader. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Running, Role: LEADER
I20260620 19:12:39.721441 21959 consensus_queue.cc:237] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:39.721925 21445 catalog_manager.cc:5495] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194). New cstate: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } health_report { overall_health: UNKNOWN } } }
I20260620 19:12:39.731766 21882 raft_consensus.cc:1261] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.731815 21602 raft_consensus.cc:1261] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:39.731942 21959 consensus_queue.cc:1035] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:39.732050 21980 consensus_queue.cc:1035] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:12:39.735653 21975 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:12:41.251663 21445 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:41.251757 21445 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:41.252441 21445 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Sending DeleteTablet for 3 replicas of tablet 9c5448e5ec544b67a68965e0262b5a49
I20260620 19:12:41.252655 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39276
I20260620 19:12:41.252682 21862 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260620 19:12:41.252878 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5448e5ec544b67a68965e0262b5a49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:40730
I20260620 19:12:41.252902 22022 tablet_replica.cc:330] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: stopping tablet replica
I20260620 19:12:41.252949 22022 raft_consensus.cc:2229] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:41.252980 22022 raft_consensus.cc:2258] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.253196 22022 ts_tablet_manager.cc:1905] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.254402 22023 tablet_replica.cc:330] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: stopping tablet replica
I20260620 19:12:41.254482 22023 raft_consensus.cc:2229] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:41.254529 22023 raft_consensus.cc:2258] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.254552 22022 ts_tablet_manager.cc:1918] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:12:41.254719 22022 log.cc:1196] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/9c5448e5ec544b67a68965e0262b5a49
I20260620 19:12:41.254741 22023 ts_tablet_manager.cc:1905] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.254945 22022 ts_tablet_manager.cc:1939] T 9c5448e5ec544b67a68965e0262b5a49 P d093f700c34f4d9a86a712d5e3971a86: Deleting consensus metadata
I20260620 19:12:41.255223 21431 catalog_manager.cc:4841] TS d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855): tablet 9c5448e5ec544b67a68965e0262b5a49 (table test-table [id=86f276d9c41b43fd9b8118d0c9601020]) successfully deleted
I20260620 19:12:41.255257 22024 tablet_replica.cc:330] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: stopping tablet replica
I20260620 19:12:41.255311 22024 raft_consensus.cc:2229] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:41.255373 22024 raft_consensus.cc:2258] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.256016 22023 ts_tablet_manager.cc:1918] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:12:41.256065 22023 log.cc:1196] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/9c5448e5ec544b67a68965e0262b5a49
I20260620 19:12:41.256232 22023 ts_tablet_manager.cc:1939] T 9c5448e5ec544b67a68965e0262b5a49 P 89c3aba1ab6446648945040d0c70282e: Deleting consensus metadata
I20260620 19:12:41.256536 21431 catalog_manager.cc:4841] TS 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697): tablet 9c5448e5ec544b67a68965e0262b5a49 (table test-table [id=86f276d9c41b43fd9b8118d0c9601020]) successfully deleted
I20260620 19:12:41.256896 22024 ts_tablet_manager.cc:1905] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.257963 22024 ts_tablet_manager.cc:1918] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:12:41.258008 22024 log.cc:1196] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/9c5448e5ec544b67a68965e0262b5a49
I20260620 19:12:41.258152 22024 ts_tablet_manager.cc:1939] T 9c5448e5ec544b67a68965e0262b5a49 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting consensus metadata
I20260620 19:12:41.258445 21429 catalog_manager.cc:4841] TS 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309): tablet 9c5448e5ec544b67a68965e0262b5a49 (table test-table [id=86f276d9c41b43fd9b8118d0c9601020]) successfully deleted
Jun 20 19:12:41 dist-test-slave-nq8r krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:12:41 dist-test-slave-nq8r krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:41.276057 22032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.269325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.19.66.254:32963)
0620 19:12:41.269462 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.269465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.269473 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:12:41.269548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.269548 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.269561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.269614 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:41.270136 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.270638 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.271333 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.271532 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.271585 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.273749 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:41.273751 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:41.273751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:41.273758 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:41.275077 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:41.275365 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:41.275366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:41.275367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:41.275417 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:41.275578 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:41.275579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:41.275579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:41.275680 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:41.275765 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.275874 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.275954 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260620 19:12:41.277441 21445 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:41.277583 21445 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:41.278775 21582 tablet_service.cc:1467] Processing CreateTablet for tablet b93d9edc360541ccb4e2895168cf0261 (DEFAULT_TABLE table=test-table [id=aa15c516601246939e9fe9eccfd389a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.278788 21722 tablet_service.cc:1467] Processing CreateTablet for tablet b93d9edc360541ccb4e2895168cf0261 (DEFAULT_TABLE table=test-table [id=aa15c516601246939e9fe9eccfd389a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.278802 21862 tablet_service.cc:1467] Processing CreateTablet for tablet b93d9edc360541ccb4e2895168cf0261 (DEFAULT_TABLE table=test-table [id=aa15c516601246939e9fe9eccfd389a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.278908 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93d9edc360541ccb4e2895168cf0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.278908 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93d9edc360541ccb4e2895168cf0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.278918 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b93d9edc360541ccb4e2895168cf0261. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.279951 22033 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Bootstrap starting.
I20260620 19:12:41.280210 22034 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Bootstrap starting.
I20260620 19:12:41.280344 22035 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Bootstrap starting.
I20260620 19:12:41.280423 22033 tablet_bootstrap.cc:654] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.280578 22034 tablet_bootstrap.cc:654] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.280676 22035 tablet_bootstrap.cc:654] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.281159 22034 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: No bootstrap required, opened a new log
I20260620 19:12:41.281193 22033 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: No bootstrap required, opened a new log
I20260620 19:12:41.281206 22034 ts_tablet_manager.cc:1397] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:41.281235 22033 ts_tablet_manager.cc:1397] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:41.281337 22034 raft_consensus.cc:348] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281364 22033 raft_consensus.cc:348] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281402 22035 tablet_bootstrap.cc:492] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: No bootstrap required, opened a new log
I20260620 19:12:41.281414 22034 raft_consensus.cc:374] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.281415 22033 raft_consensus.cc:374] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.281435 22034 raft_consensus.cc:729] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.281435 22033 raft_consensus.cc:729] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.281440 22035 ts_tablet_manager.cc:1397] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:12:41.281479 22033 consensus_queue.cc:260] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281481 22034 consensus_queue.cc:260] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281572 22034 ts_tablet_manager.cc:1428] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:41.281602 22035 raft_consensus.cc:348] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281636 22033 ts_tablet_manager.cc:1428] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:41.281670 22035 raft_consensus.cc:374] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.281689 22035 raft_consensus.cc:729] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d093f700c34f4d9a86a712d5e3971a86, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.281749 22035 consensus_queue.cc:260] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.281909 22035 ts_tablet_manager.cc:1428] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:41.291752 22039 raft_consensus.cc:482] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:41.291832 22039 raft_consensus.cc:504] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.292143 22039 leader_election.cc:288] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309)
I20260620 19:12:41.295506 22042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.292279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42309 (local address 127.19.66.195:50629)
0620 19:12:41.292456 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.292467 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:41.292513 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:41.292656 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:41.292657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:41.292659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:41.292724 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.292725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.293535 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.293536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.294654 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.294656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.295289 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.295290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.295351 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.295355 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:41.295408 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:12:41.295441 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"spinlock_wait_cycles":768,"thread_start_us":109,"threads_started":1}
I20260620 19:12:41.295526 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.292366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:50629 (local address 127.19.66.194:42309)
0620 19:12:41.292507 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.292508 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.292518 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:12:41.292568 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.292568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.292577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.292649 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:41.292790 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.293464 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.294729 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.295224 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.295295 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.295304 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.295415 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.295452 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260620 19:12:41.295773 21742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93d9edc360541ccb4e2895168cf0261" candidate_uuid: "89c3aba1ab6446648945040d0c70282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" is_pre_election: true
I20260620 19:12:41.295933 21742 raft_consensus.cc:2454] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c3aba1ab6446648945040d0c70282e in term 0.
I20260620 19:12:41.295974 22041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.292280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:42855 (local address 127.19.66.195:37183)
0620 19:12:41.292558 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.292567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:41.292611 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:41.292722 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:41.292722 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:41.292723 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:41.293070 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.293072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.293837 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.293838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.294891 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.294893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.295689 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.295691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.295786 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.295790 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:41.295838 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:12:41.295899 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":2,"thread_start_us":118,"threads_started":1}
I20260620 19:12:41.296118 21813 leader_election.cc:302] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:41.295989 22044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.292336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:37183 (local address 127.19.66.193:42855)
0620 19:12:41.292519 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.292521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.292616 (+ 95us) server_negotiation.cc:407] Connection header received
0620 19:12:41.292646 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.292646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.292655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.292718 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:41.293139 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.293767 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.294964 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.295530 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.295658 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.295667 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.295874 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.295917 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":109,"threads_started":1}
I20260620 19:12:41.296245 22039 raft_consensus.cc:2781] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:41.296277 22039 raft_consensus.cc:482] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:41.296298 22039 raft_consensus.cc:3037] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.296492 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93d9edc360541ccb4e2895168cf0261" candidate_uuid: "89c3aba1ab6446648945040d0c70282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86" is_pre_election: true
I20260620 19:12:41.296554 21602 raft_consensus.cc:2454] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89c3aba1ab6446648945040d0c70282e in term 0.
I20260620 19:12:41.297135 22039 raft_consensus.cc:504] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.297254 22039 leader_election.cc:288] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [CANDIDATE]: Term 1 election: Requested vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309)
I20260620 19:12:41.297386 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93d9edc360541ccb4e2895168cf0261" candidate_uuid: "89c3aba1ab6446648945040d0c70282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86"
I20260620 19:12:41.297403 21742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b93d9edc360541ccb4e2895168cf0261" candidate_uuid: "89c3aba1ab6446648945040d0c70282e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f7a3f9842e44500b3d08ec805dfbefc"
I20260620 19:12:41.297456 21602 raft_consensus.cc:3037] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.297464 21742 raft_consensus.cc:3037] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.298072 21602 raft_consensus.cc:2454] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c3aba1ab6446648945040d0c70282e in term 1.
I20260620 19:12:41.298141 21742 raft_consensus.cc:2454] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89c3aba1ab6446648945040d0c70282e in term 1.
I20260620 19:12:41.298242 21815 leader_election.cc:302] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89c3aba1ab6446648945040d0c70282e, d093f700c34f4d9a86a712d5e3971a86; no voters:
I20260620 19:12:41.298341 22039 raft_consensus.cc:2781] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:41.298480 22039 raft_consensus.cc:686] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 1 LEADER]: Becoming Leader. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Running, Role: LEADER
I20260620 19:12:41.298542 22039 consensus_queue.cc:237] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.299144 21444 catalog_manager.cc:5495] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e reported cstate change: term changed from 0 to 1, leader changed from <none> to 89c3aba1ab6446648945040d0c70282e (127.19.66.195). New cstate: current_term: 1 leader_uuid: "89c3aba1ab6446648945040d0c70282e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } health_report { overall_health: HEALTHY } } }
I20260620 19:12:41.305236 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.303187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.19.66.195:36697)
0620 19:12:41.303316 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.303318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.303323 (+ 5us) server_negotiation.cc:407] Connection header received
0620 19:12:41.303348 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.303349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.303363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.303428 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:41.303505 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.303959 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.304622 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.304810 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.304870 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.305002 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.305094 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.305146 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.305162 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.305175 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260620 19:12:41.309262 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.307178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:44055 (local address 127.19.66.194:42309)
0620 19:12:41.307235 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.307237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.307316 (+ 79us) server_negotiation.cc:407] Connection header received
0620 19:12:41.307360 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.307361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.307371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.307419 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:41.307529 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.307964 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.308679 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.308878 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.308930 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.309069 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.309100 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.309141 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.309183 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.309201 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:12:41.309290 22051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.307113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42309 (local address 127.19.66.195:44055)
0620 19:12:41.307256 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.307265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:41.307316 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:41.307422 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:41.307423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:41.307424 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:41.307476 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.307477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.308025 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.308026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.308602 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.308603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.308928 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.308929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.308987 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.308998 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.309140 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.309142 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:41.309182 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 19:12:41.309217 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260620 19:12:41.310931 21602 raft_consensus.cc:1261] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Refusing update from remote peer 89c3aba1ab6446648945040d0c70282e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:41.311002 21742 raft_consensus.cc:1261] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Refusing update from remote peer 89c3aba1ab6446648945040d0c70282e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:41.311131 22039 consensus_queue.cc:1035] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:41.311239 22045 consensus_queue.cc:1035] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:41.317266 22048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.315309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:59177 (local address 127.19.66.195:36697)
0620 19:12:41.315342 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.315343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.315428 (+ 85us) server_negotiation.cc:407] Connection header received
0620 19:12:41.315481 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.315482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.315491 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.315536 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:41.315603 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.316018 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.316663 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.316878 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.316930 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.317060 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.317090 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.317135 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.317185 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.317201 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:12:41.317287 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.315264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:36697 (local address 127.19.66.194:59177)
0620 19:12:41.315371 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:41.315379 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:41.315437 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:41.315537 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:41.315538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:41.315539 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:12:41.315570 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.315571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.316077 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.316078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.316612 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:41.316614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.316934 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.316935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:41.316992 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.316996 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.317140 (+ 144us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.317142 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:41.317181 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:12:41.317220 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260620 19:12:41.323880 21444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:41.323987 21444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:41.324538 21444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Sending DeleteTablet for 3 replicas of tablet b93d9edc360541ccb4e2895168cf0261
I20260620 19:12:41.324757 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet b93d9edc360541ccb4e2895168cf0261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:40730
I20260620 19:12:41.324815 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet b93d9edc360541ccb4e2895168cf0261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39276
I20260620 19:12:41.324837 22024 tablet_replica.cc:330] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: stopping tablet replica
I20260620 19:12:41.324911 22024 raft_consensus.cc:2229] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:41.324921 22022 tablet_replica.cc:330] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: stopping tablet replica
I20260620 19:12:41.324952 22024 raft_consensus.cc:2258] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.325006 21862 tablet_service.cc:1514] Processing DeleteTablet for tablet b93d9edc360541ccb4e2895168cf0261 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:41 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260620 19:12:41.325062 22022 raft_consensus.cc:2229] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:41.325073 22023 tablet_replica.cc:330] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: stopping tablet replica
I20260620 19:12:41.325105 22022 raft_consensus.cc:2258] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.325109 22023 raft_consensus.cc:2229] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:41.325171 22023 raft_consensus.cc:2258] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:41.325382 22022 ts_tablet_manager.cc:1905] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.325449 22023 ts_tablet_manager.cc:1905] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.325451 22024 ts_tablet_manager.cc:1905] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:41.326926 22023 ts_tablet_manager.cc:1918] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:41.326942 22024 ts_tablet_manager.cc:1918] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:41.326938 22022 ts_tablet_manager.cc:1918] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:12:41.326974 22023 log.cc:1196] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/b93d9edc360541ccb4e2895168cf0261
I20260620 19:12:41.326987 22022 log.cc:1196] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/b93d9edc360541ccb4e2895168cf0261
I20260620 19:12:41.326987 22024 log.cc:1196] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/b93d9edc360541ccb4e2895168cf0261
I20260620 19:12:41.327173 22023 ts_tablet_manager.cc:1939] T b93d9edc360541ccb4e2895168cf0261 P 89c3aba1ab6446648945040d0c70282e: Deleting consensus metadata
I20260620 19:12:41.327193 22022 ts_tablet_manager.cc:1939] T b93d9edc360541ccb4e2895168cf0261 P d093f700c34f4d9a86a712d5e3971a86: Deleting consensus metadata
I20260620 19:12:41.327210 22024 ts_tablet_manager.cc:1939] T b93d9edc360541ccb4e2895168cf0261 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting consensus metadata
I20260620 19:12:41.327414 21444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 19:12:41.327488 21429 catalog_manager.cc:4841] TS 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309): tablet b93d9edc360541ccb4e2895168cf0261 (table test-table [id=aa15c516601246939e9fe9eccfd389a4]) successfully deleted
W20260620 19:12:41.327554 21444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:41.327498 21431 catalog_manager.cc:4841] TS d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855): tablet b93d9edc360541ccb4e2895168cf0261 (table test-table [id=aa15c516601246939e9fe9eccfd389a4]) successfully deleted
I20260620 19:12:41.327759 21431 catalog_manager.cc:4841] TS 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697): tablet b93d9edc360541ccb4e2895168cf0261 (table test-table [id=aa15c516601246939e9fe9eccfd389a4]) successfully deleted
I20260620 19:12:41.328540 21862 tablet_service.cc:1467] Processing CreateTablet for tablet 9483f1db36084b4bb6486221aaa559e9 (DEFAULT_TABLE table=test-table [id=404602241e064a3fac25860708f974f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.328547 21582 tablet_service.cc:1467] Processing CreateTablet for tablet 9483f1db36084b4bb6486221aaa559e9 (DEFAULT_TABLE table=test-table [id=404602241e064a3fac25860708f974f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.328644 21582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9483f1db36084b4bb6486221aaa559e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.328644 21862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9483f1db36084b4bb6486221aaa559e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.329474 22033 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Bootstrap starting.
I20260620 19:12:41.329492 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 9483f1db36084b4bb6486221aaa559e9 (DEFAULT_TABLE table=test-table [id=404602241e064a3fac25860708f974f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:41.329571 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9483f1db36084b4bb6486221aaa559e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:41.329723 22035 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86: Bootstrap starting.
I20260620 19:12:41.329818 22033 tablet_bootstrap.cc:654] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.330262 22035 tablet_bootstrap.cc:654] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.330479 22033 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: No bootstrap required, opened a new log
I20260620 19:12:41.330525 22033 ts_tablet_manager.cc:1397] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:41.330696 22034 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Bootstrap starting.
I20260620 19:12:41.330690 22033 raft_consensus.cc:348] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.330760 22033 raft_consensus.cc:374] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.330780 22033 raft_consensus.cc:729] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c3aba1ab6446648945040d0c70282e, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.330821 22033 consensus_queue.cc:260] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.330832 22035 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86: No bootstrap required, opened a new log
I20260620 19:12:41.330873 22035 ts_tablet_manager.cc:1397] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 19:12:41.330925 22033 ts_tablet_manager.cc:1428] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:41.330989 22034 tablet_bootstrap.cc:654] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:41.331000 22035 raft_consensus.cc:348] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.331059 22035 raft_consensus.cc:374] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.331079 22035 raft_consensus.cc:729] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d093f700c34f4d9a86a712d5e3971a86, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.331120 22035 consensus_queue.cc:260] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.331212 22035 ts_tablet_manager.cc:1428] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:41.331672 22034 tablet_bootstrap.cc:492] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: No bootstrap required, opened a new log
I20260620 19:12:41.331739 22034 ts_tablet_manager.cc:1397] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:12:41.331831 22034 raft_consensus.cc:348] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.331893 22034 raft_consensus.cc:374] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:41.331918 22034 raft_consensus.cc:729] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Initialized, Role: FOLLOWER
I20260620 19:12:41.331961 22034 consensus_queue.cc:260] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.332042 22034 ts_tablet_manager.cc:1428] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:41.574819 22017 raft_consensus.cc:482] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:41.574924 22017 raft_consensus.cc:504] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.575102 22017 leader_election.cc:288] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:41.575295 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9483f1db36084b4bb6486221aaa559e9" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86" is_pre_election: true
I20260620 19:12:41.575342 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9483f1db36084b4bb6486221aaa559e9" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e" is_pre_election: true
I20260620 19:12:41.575392 21602 raft_consensus.cc:2454] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:41.575414 21882 raft_consensus.cc:2454] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 0.
I20260620 19:12:41.575543 21675 leader_election.cc:302] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, d093f700c34f4d9a86a712d5e3971a86; no voters:
I20260620 19:12:41.575629 22017 raft_consensus.cc:2781] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:41.575667 22017 raft_consensus.cc:482] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:41.575683 22017 raft_consensus.cc:3037] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.576336 22017 raft_consensus.cc:504] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.576436 22017 leader_election.cc:288] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Requested vote from peers d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855), 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697)
I20260620 19:12:41.576623 21882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9483f1db36084b4bb6486221aaa559e9" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89c3aba1ab6446648945040d0c70282e"
I20260620 19:12:41.576613 21602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9483f1db36084b4bb6486221aaa559e9" candidate_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d093f700c34f4d9a86a712d5e3971a86"
I20260620 19:12:41.576694 21882 raft_consensus.cc:3037] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.576701 21602 raft_consensus.cc:3037] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:41.577307 21882 raft_consensus.cc:2454] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:41.577358 21602 raft_consensus.cc:2454] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f7a3f9842e44500b3d08ec805dfbefc in term 1.
I20260620 19:12:41.577468 21675 leader_election.cc:302] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f7a3f9842e44500b3d08ec805dfbefc, 89c3aba1ab6446648945040d0c70282e; no voters:
I20260620 19:12:41.577562 22017 raft_consensus.cc:2781] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:41.577613 22017 raft_consensus.cc:686] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Becoming Leader. State: Replica: 4f7a3f9842e44500b3d08ec805dfbefc, State: Running, Role: LEADER
I20260620 19:12:41.577664 22017 consensus_queue.cc:237] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } }
I20260620 19:12:41.578217 21444 catalog_manager.cc:5495] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194). New cstate: current_term: 1 leader_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f7a3f9842e44500b3d08ec805dfbefc" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 } health_report { overall_health: UNKNOWN } } }
I20260620 19:12:41.616250 22043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:41.614409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40764 (local address 127.19.66.194:42309)
0620 19:12:41.614451 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:41.614452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:41.614461 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:41.614514 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:41.614514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:41.614524 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:41.614573 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:41.614661 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.615117 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.615772 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:41.615953 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:41.616000 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:41.616084 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:41.616108 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:41.616139 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:41.616179 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 19:12:41.616195 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:12:41.618104 21882 raft_consensus.cc:1261] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:41.618161 21602 raft_consensus.cc:1261] T 9483f1db36084b4bb6486221aaa559e9 P d093f700c34f4d9a86a712d5e3971a86 [term 1 FOLLOWER]: Refusing update from remote peer 4f7a3f9842e44500b3d08ec805dfbefc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:41.618312 22017 consensus_queue.cc:1035] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d093f700c34f4d9a86a712d5e3971a86" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:41.618423 22009 consensus_queue.cc:1035] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [LEADER]: Connected to new peer: Peer: permanent_uuid: "89c3aba1ab6446648945040d0c70282e" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:12:41.621994 22002 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 19:12:43.139219 21444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:43.139317 21444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:43.140061 21444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4ef0ec1c279d4631bca48804f372a971: Sending DeleteTablet for 3 replicas of tablet 9483f1db36084b4bb6486221aaa559e9
I20260620 19:12:43.140285 21862 tablet_service.cc:1514] Processing DeleteTablet for tablet 9483f1db36084b4bb6486221aaa559e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:43 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:34124
I20260620 19:12:43.140311 21722 tablet_service.cc:1514] Processing DeleteTablet for tablet 9483f1db36084b4bb6486221aaa559e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:43 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:40730
I20260620 19:12:43.140309 21582 tablet_service.cc:1514] Processing DeleteTablet for tablet 9483f1db36084b4bb6486221aaa559e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:43 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39276
I20260620 19:12:43.140475 22084 tablet_replica.cc:330] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: stopping tablet replica
I20260620 19:12:43.140494 22086 tablet_replica.cc:330] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: stopping tablet replica
I20260620 19:12:43.140543 22086 raft_consensus.cc:2229] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:43.140604 22086 raft_consensus.cc:2258] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:43.140594 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21517
I20260620 19:12:43.141860 22086 ts_tablet_manager.cc:1905] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:43.143210 22086 ts_tablet_manager.cc:1918] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:12:43.143260 22086 log.cc:1196] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/9483f1db36084b4bb6486221aaa559e9
I20260620 19:12:43.143481 22086 ts_tablet_manager.cc:1939] T 9483f1db36084b4bb6486221aaa559e9 P 4f7a3f9842e44500b3d08ec805dfbefc: Deleting consensus metadata
I20260620 19:12:43.143689 22084 raft_consensus.cc:2229] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:43.143778 22084 raft_consensus.cc:2258] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:43.143760 21429 catalog_manager.cc:4841] TS 4f7a3f9842e44500b3d08ec805dfbefc (127.19.66.194:42309): tablet 9483f1db36084b4bb6486221aaa559e9 (table test-table [id=404602241e064a3fac25860708f974f7]) successfully deleted
I20260620 19:12:43.144011 22084 ts_tablet_manager.cc:1905] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:43.145125 22084 ts_tablet_manager.cc:1918] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 19:12:43.145175 22084 log.cc:1196] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/9483f1db36084b4bb6486221aaa559e9
I20260620 19:12:43.145347 22084 ts_tablet_manager.cc:1939] T 9483f1db36084b4bb6486221aaa559e9 P 89c3aba1ab6446648945040d0c70282e: Deleting consensus metadata
W20260620 19:12:43.145707 21431 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:42855 (error 108)
I20260620 19:12:43.145872 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21657
I20260620 19:12:43.145951 21431 catalog_manager.cc:4841] TS 89c3aba1ab6446648945040d0c70282e (127.19.66.195:36697): tablet 9483f1db36084b4bb6486221aaa559e9 (table test-table [id=404602241e064a3fac25860708f974f7]) successfully deleted
I20260620 19:12:43.146878 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:43.146682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:42855 (local address 127.0.0.1:39292)
0620 19:12:43.146789 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:43.146838 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:42855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
W20260620 19:12:43.146973 21431 catalog_manager.cc:4584] TS d093f700c34f4d9a86a712d5e3971a86 (127.19.66.193:42855): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9483f1db36084b4bb6486221aaa559e9: Network error: Client connection negotiation failed: client connection to 127.19.66.193:42855: connect: Connection refused (error 111)
I20260620 19:12:43.151234 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21797
I20260620 19:12:43.156061 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 21413
2026-06-20T19:12:43Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5707 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:43 dist-test-slave-nq8r krb5kdc[22091](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:43 dist-test-slave-nq8r krb5kdc[22091](info): set up 2 sockets
Jun 20 19:12:43 dist-test-slave-nq8r krb5kdc[22091](info): commencing operation
krb5kdc: starting...
W20260620 19:12:44.100368 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:44.205406 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37393
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:37393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:44.283798 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:44.284029 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:44.284061 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:44.285262 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:44.285305 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:44.285326 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:44.285341 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:44.285355 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:44.286821 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:37393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:44.287065 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:44.287269 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:44.291821 22110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.291848 22112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:44.291860 22104 server_base.cc:1034] running on GCE node
W20260620 19:12:44.291828 22109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:44.292299 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:44.292599 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:44.293757 22104 hybrid_clock.cc:648] HybridClock initialized: now 1781982764293733 us; error 34 us; skew 500 ppm
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:44.296140 22104 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:44.296550 22104 webserver.cc:460] Webserver started at http://127.19.66.254:42141/ using document root <none> and password file <none>
I20260620 19:12:44.296725 22104 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:44.296762 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:44.296883 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:44.297897 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "7c384c0da8d6451d8b22484b157c4978"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "d488041aec53812d47fa03591f469158"
server_key_iv: "4492f38a00eb42d144ae4061a14063a6"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.298188 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "7c384c0da8d6451d8b22484b157c4978"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "d488041aec53812d47fa03591f469158"
server_key_iv: "4492f38a00eb42d144ae4061a14063a6"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.299301 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:44.299911 22119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.300076 22104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:44.300146 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "7c384c0da8d6451d8b22484b157c4978"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "d488041aec53812d47fa03591f469158"
server_key_iv: "4492f38a00eb42d144ae4061a14063a6"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.300222 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:44.315027 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:44.316947 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:44.317070 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:44.320604 22104 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:37393
I20260620 19:12:44.320648 22171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:37393 every 8 connection(s)
I20260620 19:12:44.321008 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:44.321882 22172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:44.324047 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978: Bootstrap starting.
I20260620 19:12:44.324594 22172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:44.324863 22172 log.cc:824] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:44.325485 22172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978: No bootstrap required, opened a new log
I20260620 19:12:44.326992 22172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } }
I20260620 19:12:44.327127 22172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:44.327149 22172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c384c0da8d6451d8b22484b157c4978, State: Initialized, Role: FOLLOWER
I20260620 19:12:44.327291 22172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } }
I20260620 19:12:44.327351 22172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:44.327373 22172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:44.327414 22172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:44.327934 22172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } }
I20260620 19:12:44.328022 22172 leader_election.cc:302] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c384c0da8d6451d8b22484b157c4978; no voters:
I20260620 19:12:44.328173 22172 leader_election.cc:288] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:44.328212 22175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:44.328351 22175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [term 1 LEADER]: Becoming Leader. State: Replica: 7c384c0da8d6451d8b22484b157c4978, State: Running, Role: LEADER
I20260620 19:12:44.328403 22172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:44.328466 22175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } }
I20260620 19:12:44.328814 22176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c384c0da8d6451d8b22484b157c4978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } } }
I20260620 19:12:44.328925 22176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:44.328819 22177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c384c0da8d6451d8b22484b157c4978. Latest consensus state: current_term: 1 leader_uuid: "7c384c0da8d6451d8b22484b157c4978" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c384c0da8d6451d8b22484b157c4978" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37393 } } }
I20260620 19:12:44.329010 22177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:44.329154 22179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:44.329478 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22104
I20260620 19:12:44.329559 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:44.329701 22179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:44.329717 19723 external_mini_cluster.cc:1428] Setting key fea22e30c679ab076dd02973356cbb72
I20260620 19:12:44.333266 22179 catalog_manager.cc:1347] Generated new cluster ID: c3135790140c400191fe63a7e48ea416
I20260620 19:12:44.333315 22179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:44.340976 22179 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:44.341578 22179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:44.343839 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.330360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.19.66.254:37393)
0620 19:12:44.330512 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:44.330524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:44.330559 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:12:44.330712 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:44.330719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:44.330780 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:44.330916 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:44.331522 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.332425 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.336914 (+ 4489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.337521 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.337655 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.340250 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:44.340255 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:44.340255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:44.340281 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:44.342319 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.342678 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.342681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.342681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.342792 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.342963 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.342964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.342964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.343243 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:44.343402 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:44.343466 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:12:44.343626 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 19:12:44.351430 22179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c384c0da8d6451d8b22484b157c4978: Generated new TSK 0
I20260620 19:12:44.351644 22179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:44.375428 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:44.453456 22198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:44.453656 22198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:44.453680 22198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:44.454803 22198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:44.454841 22198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:44.454867 22198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:44.456260 22198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:44.456482 22198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:44.456687 22198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:44.461161 22206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.461187 22203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.461314 22204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:44.461614 22198 server_base.cc:1034] running on GCE node
I20260620 19:12:44.461822 22198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:44.462039 22198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:44.463202 22198 hybrid_clock.cc:648] HybridClock initialized: now 1781982764463183 us; error 32 us; skew 500 ppm
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:44.465541 22198 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:44.465943 22198 webserver.cc:460] Webserver started at http://127.19.66.193:35559/ using document root <none> and password file <none>
I20260620 19:12:44.466150 22198 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:44.466197 22198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:44.466317 22198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:44.467319 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "c48f39b59e0d4cd2b83e72b2f9982c98"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "19f5cc605a1cc3873cc34ef02f1bad0b"
server_key_iv: "383cc3ac896232aefd09428a5049220e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.467641 22198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "c48f39b59e0d4cd2b83e72b2f9982c98"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "19f5cc605a1cc3873cc34ef02f1bad0b"
server_key_iv: "383cc3ac896232aefd09428a5049220e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.468878 22198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:44.469620 22213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.469770 22198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:44.469861 22198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "c48f39b59e0d4cd2b83e72b2f9982c98"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "19f5cc605a1cc3873cc34ef02f1bad0b"
server_key_iv: "383cc3ac896232aefd09428a5049220e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.469928 22198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:44.484436 22198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:44.486282 22198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:44.486418 22198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:44.486637 22198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:44.486948 22198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:44.486982 22198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.487011 22198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:44.487028 22198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.492532 22198 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:42027
I20260620 19:12:44.492569 22326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:42027 every 8 connection(s)
I20260620 19:12:44.492894 22198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:44.499697 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22198
I20260620 19:12:44.499778 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:44.499929 19723 external_mini_cluster.cc:1428] Setting key 33dfe64a7036e9ad16e964da05318721
I20260620 19:12:44.502004 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.493576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:55635 (local address 127.19.66.254:37393)
0620 19:12:44.493655 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:44.493658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:44.493836 (+ 178us) server_negotiation.cc:407] Connection header received
0620 19:12:44.494284 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:44.494285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:44.494297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:44.494353 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:44.495247 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.495755 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.496720 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.496963 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.497013 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.499028 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:44.499029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:44.499030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:44.499036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:44.500612 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.501065 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.501066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.501066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.501120 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.501359 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.501359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.501359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.501469 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:44.501529 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:44.501821 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:12:44.501911 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:12:44.502215 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.493557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37393 (local address 127.19.66.193:55635)
0620 19:12:44.493697 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.493733 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:44.494206 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:44.494406 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:44.494410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:44.494962 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:44.495156 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.495162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.495823 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.495824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.496651 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.496653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.497032 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.497033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.497141 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.497666 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:44.497675 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:44.498939 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:44.500685 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.500692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.500694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.500996 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.501174 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.501175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.501175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.501306 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.501524 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:44.501529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:44.501703 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 19:12:44.501810 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:12:44.501980 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260620 19:12:44.502790 22327 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37393
I20260620 19:12:44.502884 22327 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:44.503083 22327 heartbeater.cc:511] Master 127.19.66.254:37393 requested a full tablet report, sending...
I20260620 19:12:44.503604 22136 ts_manager.cc:194] Registered new tserver with Master: c48f39b59e0d4cd2b83e72b2f9982c98 (127.19.66.193:42027)
I20260620 19:12:44.504345 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:55635
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:44.521384 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:44.599947 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:44.600174 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:44.600209 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:44.601393 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:44.601432 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:44.601466 22332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:44.602851 22332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:44.603091 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:44.603289 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:44.607971 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.608008 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.607978 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:44.608234 22332 server_base.cc:1034] running on GCE node
I20260620 19:12:44.608424 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:44.608632 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:44.609772 22332 hybrid_clock.cc:648] HybridClock initialized: now 1781982764609757 us; error 28 us; skew 500 ppm
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:44.612033 22332 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:44.612404 22332 webserver.cc:460] Webserver started at http://127.19.66.194:43659/ using document root <none> and password file <none>
I20260620 19:12:44.612567 22332 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:44.612604 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:44.612699 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:44.613709 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "68473d2c364b46218ec2404810e2b44b"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "8796002266d1280f6f93c517371d2b00"
server_key_iv: "b2d5855306fc9a9a2ed5b4d499ac268e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.613988 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "68473d2c364b46218ec2404810e2b44b"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "8796002266d1280f6f93c517371d2b00"
server_key_iv: "b2d5855306fc9a9a2ed5b4d499ac268e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.615175 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:44.615790 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.615928 22332 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:44.616020 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "68473d2c364b46218ec2404810e2b44b"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "8796002266d1280f6f93c517371d2b00"
server_key_iv: "b2d5855306fc9a9a2ed5b4d499ac268e"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.616086 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:44.632067 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:44.634037 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:44.634176 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:44.634387 22332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:44.634677 22332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:44.634709 22332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.634739 22332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:44.634763 22332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.640007 22332 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:45381
I20260620 19:12:44.640049 22460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:45381 every 8 connection(s)
I20260620 19:12:44.640357 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:44.645534 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22332
I20260620 19:12:44.645632 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:44.645855 19723 external_mini_cluster.cc:1428] Setting key adbc2a084cfb022545b9ef3d1d37012a
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:44.649224 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.641108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:41887 (local address 127.19.66.254:37393)
0620 19:12:44.641184 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:44.641186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:44.641302 (+ 116us) server_negotiation.cc:407] Connection header received
0620 19:12:44.641758 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:44.641759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:44.641769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:44.641816 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:44.642684 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.643158 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.644153 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.644330 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.644373 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.646534 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:44.646535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:44.646535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:44.646543 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:44.647882 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.648358 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.648359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.648359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.648409 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.648667 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.648668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.648668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.648766 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:44.648811 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:44.649097 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:12:44.649162 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:12:44.649417 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.641038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37393 (local address 127.19.66.194:41887)
0620 19:12:44.641184 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.641220 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:44.641681 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:44.641873 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:44.641876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:44.642429 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:44.642619 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.642624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.643236 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.643237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.644089 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.644091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.644382 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.644383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.644486 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.644986 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:44.644994 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:44.646456 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:44.647969 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.647976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.647978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.648273 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.648462 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.648463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.648463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.648601 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.648818 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:44.648823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:44.648976 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 19:12:44.649085 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 19:12:44.649210 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260620 19:12:44.649864 22461 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37393
I20260620 19:12:44.649961 22461 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:44.650143 22461 heartbeater.cc:511] Master 127.19.66.254:37393 requested a full tablet report, sending...
I20260620 19:12:44.650489 22136 ts_manager.cc:194] Registered new tserver with Master: 68473d2c364b46218ec2404810e2b44b (127.19.66.194:45381)
I20260620 19:12:44.650905 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:41887
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:44.668593 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:44.747051 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:44.747267 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:44.747296 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:44.748441 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:44.748481 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:44.748517 22466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:44.749943 22466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:44.750173 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:44.750381 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:44.754964 22474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.754980 22472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:44.754964 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:44.755251 22466 server_base.cc:1034] running on GCE node
I20260620 19:12:44.755448 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:44.755692 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:44.756832 22466 hybrid_clock.cc:648] HybridClock initialized: now 1781982764756823 us; error 36 us; skew 500 ppm
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:44.759253 22466 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:44.759689 22466 webserver.cc:460] Webserver started at http://127.19.66.195:44581/ using document root <none> and password file <none>
I20260620 19:12:44.759891 22466 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:44.759940 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:44.760066 22466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:44.761152 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "6be320345842456888a6c79613e1cff5"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "955bea94493401987f3a5f1854314285"
server_key_iv: "26e8091113b211b2aea6d2ce16e1e05d"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.761476 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "6be320345842456888a6c79613e1cff5"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "955bea94493401987f3a5f1854314285"
server_key_iv: "26e8091113b211b2aea6d2ce16e1e05d"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.762651 22466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:44.763280 22481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.763430 22466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:44.763511 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "6be320345842456888a6c79613e1cff5"
format_stamp: "Formatted at 2026-06-20 19:12:44 on dist-test-slave-nq8r"
server_key: "955bea94493401987f3a5f1854314285"
server_key_iv: "26e8091113b211b2aea6d2ce16e1e05d"
server_key_version: "encryptionkey@0"
I20260620 19:12:44.763567 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:44.777280 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:44.779804 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:44.779942 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:44.780267 22466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:44.780611 22466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:44.780653 22466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.780682 22466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:44.780699 22466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:44.786300 22466 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:44517
I20260620 19:12:44.786337 22594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:44517 every 8 connection(s)
I20260620 19:12:44.786657 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22091](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982764, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:44.792752 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22466
I20260620 19:12:44.792838 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:44.793040 19723 external_mini_cluster.cc:1428] Setting key bf71c0be631e2bb2551075327e1b68af
I20260620 19:12:44.795410 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.787325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:46369 (local address 127.19.66.254:37393)
0620 19:12:44.787385 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:44.787387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:44.787567 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:12:44.788069 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:44.788070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:44.788085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:44.788138 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:44.789111 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.789607 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.790606 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.790802 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.790858 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.792829 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:44.792830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:44.792831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:44.792838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:44.794216 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.794564 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.794566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.794566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.794621 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:44.794830 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:44.794831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:44.794832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:44.794944 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:44.794997 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:44.795248 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:12:44.795323 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:12:44.795579 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:44.787304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37393 (local address 127.19.66.195:46369)
0620 19:12:44.787453 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:44.787489 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:44.787972 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:44.788214 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:44.788217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:44.788784 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:44.789023 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.789029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.789691 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.789692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.790522 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:44.790524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:44.790871 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:44.790872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:44.790987 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:44.791459 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:44.791467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:44.792745 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:44.794287 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.794294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.794296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.794501 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.794677 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:44.794678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:44.794678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:44.794777 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:44.794996 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:44.795002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:44.795132 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 19:12:44.795222 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 19:12:44.795372 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260620 19:12:44.795995 22595 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37393
I20260620 19:12:44.796087 22595 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:44.796259 22595 heartbeater.cc:511] Master 127.19.66.254:37393 requested a full tablet report, sending...
I20260620 19:12:44.796595 22136 ts_manager.cc:194] Registered new tserver with Master: 6be320345842456888a6c79613e1cff5 (127.19.66.195:44517)
I20260620 19:12:44.797009 22136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:46369
I20260620 19:12:44.804550 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:12:44.814584 22194 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:44.813759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.19.66.254:37393)
0620 19:12:44.813819 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:44.813821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:44.813900 (+ 79us) server_negotiation.cc:407] Connection header received
0620 19:12:44.813949 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:44.813950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:44.813958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:44.814017 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:44.814511 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56906: BlockingRecv error: recv got EOF from 127.0.0.1:56906 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:12:44.815480 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22198
I20260620 19:12:44.819448 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22332
I20260620 19:12:44.823045 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22466
I20260620 19:12:44.826460 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22104
2026-06-20T19:12:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1669 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22607](info): set up 2 sockets
Jun 20 19:12:44 dist-test-slave-nq8r krb5kdc[22607](info): commencing operation
krb5kdc: starting...
W20260620 19:12:45.772466 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:45 dist-test-slave-nq8r krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:45.877730 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38077
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:45.955585 22620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:45.955788 22620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:45.955809 22620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:45.956966 22620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:45.957001 22620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:45.957013 22620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:45.957024 22620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:45.957034 22620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:45.958478 22620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:45.958669 22620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:45.958842 22620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:45.963263 22628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:45.963371 22620 server_base.cc:1034] running on GCE node
W20260620 19:12:45.963346 22626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:45.963335 22625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:45.963711 22620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:45.963965 22620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:45.965095 22620 hybrid_clock.cc:648] HybridClock initialized: now 1781982765965080 us; error 32 us; skew 500 ppm
Jun 20 19:12:45 dist-test-slave-nq8r krb5kdc[22607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982765, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:45.967424 22620 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:45.967836 22620 webserver.cc:460] Webserver started at http://127.19.66.254:40521/ using document root <none> and password file <none>
I20260620 19:12:45.968035 22620 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:45.968077 22620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:45.968200 22620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:45.969260 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "756d49f419f2424aa549dc4c27d141d0"
format_stamp: "Formatted at 2026-06-20 19:12:45 on dist-test-slave-nq8r"
server_key: "e735ff051e3df1f5d48b63541c1429e4"
server_key_iv: "e984eb6643271b6ecda47e49fb38eae1"
server_key_version: "encryptionkey@0"
I20260620 19:12:45.969573 22620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "756d49f419f2424aa549dc4c27d141d0"
format_stamp: "Formatted at 2026-06-20 19:12:45 on dist-test-slave-nq8r"
server_key: "e735ff051e3df1f5d48b63541c1429e4"
server_key_iv: "e984eb6643271b6ecda47e49fb38eae1"
server_key_version: "encryptionkey@0"
I20260620 19:12:45.970748 22620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:45.971410 22635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:45.971546 22620 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:45.971618 22620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "756d49f419f2424aa549dc4c27d141d0"
format_stamp: "Formatted at 2026-06-20 19:12:45 on dist-test-slave-nq8r"
server_key: "e735ff051e3df1f5d48b63541c1429e4"
server_key_iv: "e984eb6643271b6ecda47e49fb38eae1"
server_key_version: "encryptionkey@0"
I20260620 19:12:45.971681 22620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:45.995725 22620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:45.997954 22620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:45.998085 22620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:46.001924 22620 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:38077
I20260620 19:12:46.001991 22687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:38077 every 8 connection(s)
I20260620 19:12:46.002434 22620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:46.003099 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:46.005447 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0: Bootstrap starting.
I20260620 19:12:46.006004 22688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:46.006283 22688 log.cc:824] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:46.006953 22688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0: No bootstrap required, opened a new log
I20260620 19:12:46.008632 22688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } }
I20260620 19:12:46.008773 22688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:46.008795 22688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756d49f419f2424aa549dc4c27d141d0, State: Initialized, Role: FOLLOWER
I20260620 19:12:46.008963 22688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } }
I20260620 19:12:46.009029 22688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:46.009052 22688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:46.009127 22688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:46.010002 22688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } }
I20260620 19:12:46.010105 22688 leader_election.cc:302] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 756d49f419f2424aa549dc4c27d141d0; no voters:
I20260620 19:12:46.010262 22688 leader_election.cc:288] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:46.010298 22691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:46.010427 22691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [term 1 LEADER]: Becoming Leader. State: Replica: 756d49f419f2424aa549dc4c27d141d0, State: Running, Role: LEADER
I20260620 19:12:46.010506 22688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:46.010537 22691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } }
I20260620 19:12:46.010885 22692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "756d49f419f2424aa549dc4c27d141d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } } }
I20260620 19:12:46.010890 22693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 756d49f419f2424aa549dc4c27d141d0. Latest consensus state: current_term: 1 leader_uuid: "756d49f419f2424aa549dc4c27d141d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "756d49f419f2424aa549dc4c27d141d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38077 } } }
I20260620 19:12:46.011050 22693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:46.011395 22692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:46.011520 22700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:46.011993 22700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:46.011993 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22620
I20260620 19:12:46.012073 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:46.012238 19723 external_mini_cluster.cc:1428] Setting key cd1fd52f3417dbdffea1497e363e03ce
I20260620 19:12:46.013677 22700 catalog_manager.cc:1347] Generated new cluster ID: aa8db8d7a23a4213a68dea1edf00c975
I20260620 19:12:46.013716 22700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:46 dist-test-slave-nq8r krb5kdc[22607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:46.025339 22700 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:46.025990 22700 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:46.027580 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:46.012683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.19.66.254:38077)
0620 19:12:46.012833 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:46.012863 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:46.017147 (+ 4284us) server_negotiation.cc:407] Connection header received
0620 19:12:46.017276 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:46.017284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:46.017341 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:46.017499 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:46.018165 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:46.019623 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:46.020633 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:46.021323 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:46.021424 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:46.024076 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:46.024081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:46.024081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:46.024093 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:46.026082 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:46.026472 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:46.026475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:46.026475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:46.026589 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:46.026775 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:46.026775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:46.026775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:46.027015 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:46.027144 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:46.027237 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:12:46.027366 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260620 19:12:46.028493 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:46.031692 22700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 756d49f419f2424aa549dc4c27d141d0: Generated new TSK 0
I20260620 19:12:46.031852 22700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:46.043810 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:46.040188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55968 (local address 127.19.66.254:38077)
0620 19:12:46.040241 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:46.040243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:46.040334 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:12:46.040379 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:46.040380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:46.040390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:46.040462 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:46.040771 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:46.041571 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:46.042272 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:46.042611 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:46.042676 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:46.043329 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:46.043330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:46.043330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:46.043554 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:46.043573 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:46.043581 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:46.043650 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:46.043698 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 19:12:46.043725 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:12:46.045954 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22620
2026-06-20T19:12:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1218 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:46 dist-test-slave-nq8r krb5kdc[22721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:46 dist-test-slave-nq8r krb5kdc[22721](info): set up 2 sockets
Jun 20 19:12:46 dist-test-slave-nq8r krb5kdc[22721](info): commencing operation
krb5kdc: starting...
W20260620 19:12:46.976547 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.914s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:47.084244 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33159
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:33159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:47.162868 22734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:47.163077 22734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:47.163100 22734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:47.164197 22734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:47.164232 22734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:47.164244 22734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:47.164255 22734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:47.164268 22734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:47.165691 22734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:33159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:47.165880 22734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:47.166049 22734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:47.170722 22734 server_base.cc:1034] running on GCE node
W20260620 19:12:47.170662 22740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.170750 22742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.170671 22739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:47.171053 22734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:47.171350 22734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:47.172492 22734 hybrid_clock.cc:648] HybridClock initialized: now 1781982767172479 us; error 29 us; skew 500 ppm
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:47.177697 22734 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:47.178118 22734 webserver.cc:460] Webserver started at http://127.19.66.254:32805/ using document root <none> and password file <none>
I20260620 19:12:47.178335 22734 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:47.178383 22734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:47.178541 22734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:47.179589 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f2155d1a04c842649e5da781102df0db"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "5c453ec40770ccbdf90eead8319bf446"
server_key_iv: "3dce8a5675f93ab22c03ff2ad8cf5cca"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.179915 22734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f2155d1a04c842649e5da781102df0db"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "5c453ec40770ccbdf90eead8319bf446"
server_key_iv: "3dce8a5675f93ab22c03ff2ad8cf5cca"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.181195 22734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:47.181934 22749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.182088 22734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:47.182173 22734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f2155d1a04c842649e5da781102df0db"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "5c453ec40770ccbdf90eead8319bf446"
server_key_iv: "3dce8a5675f93ab22c03ff2ad8cf5cca"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.182231 22734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:47.197292 22734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:47.199123 22734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:47.199265 22734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:47.202955 22734 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:33159
I20260620 19:12:47.202996 22801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:33159 every 8 connection(s)
I20260620 19:12:47.203327 22734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:47.204306 22802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:47.206532 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: Bootstrap starting.
I20260620 19:12:47.207067 22802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:47.207314 22802 log.cc:824] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:47.207934 22802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: No bootstrap required, opened a new log
I20260620 19:12:47.208782 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22734
I20260620 19:12:47.208901 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:47.209067 19723 external_mini_cluster.cc:1428] Setting key 766f14ee2d5ae697d324c0f21bb1de6c
I20260620 19:12:47.209882 22802 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } }
I20260620 19:12:47.210026 22802 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:47.210059 22802 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2155d1a04c842649e5da781102df0db, State: Initialized, Role: FOLLOWER
I20260620 19:12:47.210247 22802 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } }
I20260620 19:12:47.210332 22802 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:47.210368 22802 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:47.210417 22802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:47.211336 22802 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } }
I20260620 19:12:47.211426 22802 leader_election.cc:302] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2155d1a04c842649e5da781102df0db; no voters:
I20260620 19:12:47.211555 22802 leader_election.cc:288] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:47.211596 22807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:47.211779 22807 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [term 1 LEADER]: Becoming Leader. State: Replica: f2155d1a04c842649e5da781102df0db, State: Running, Role: LEADER
I20260620 19:12:47.211787 22802 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:47.211954 22807 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } }
I20260620 19:12:47.213387 22808 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2155d1a04c842649e5da781102df0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } } }
I20260620 19:12:47.213478 22808 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:47.213640 22822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 19:12:47.213685 22809 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2155d1a04c842649e5da781102df0db. Latest consensus state: current_term: 1 leader_uuid: "f2155d1a04c842649e5da781102df0db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2155d1a04c842649e5da781102df0db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33159 } } }
I20260620 19:12:47.213769 22809 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:47.213723 22822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:47.214150 22824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:47.214320 22824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:47.215593 22824 catalog_manager.cc:1347] Generated new cluster ID: 58c6c3569123485194392c0548daae81
I20260620 19:12:47.215646 22824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:47.219885 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.209569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.19.66.254:33159)
0620 19:12:47.209725 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:47.209733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:47.209760 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:12:47.209857 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:47.209862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:47.209904 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:47.210011 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:47.210885 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.211986 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.213006 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.213577 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.213953 (+ 376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.216238 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:47.216242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:47.216242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:47.216262 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:47.218194 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.218554 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.218558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.218558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.218687 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.218860 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.218861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.218861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.219189 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:47.219378 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:47.219443 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 19:12:47.219683 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":64,"threads_started":1}
I20260620 19:12:47.232343 22824 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:47.232950 22824 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:47.239306 22824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2155d1a04c842649e5da781102df0db: Generated new TSK 0
I20260620 19:12:47.239481 22824 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:47.265533 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:47.348039 22828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:47.348235 22828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:47.348258 22828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:47.349377 22828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:47.349436 22828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:47.349462 22828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:47.351099 22828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:47.351305 22828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:47.351482 22828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:47.356187 22834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.356206 22836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.356213 22833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:47.356240 22828 server_base.cc:1034] running on GCE node
I20260620 19:12:47.356631 22828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:47.356902 22828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:47.358094 22828 hybrid_clock.cc:648] HybridClock initialized: now 1781982767358068 us; error 32 us; skew 500 ppm
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:47.360491 22828 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:47.360951 22828 webserver.cc:460] Webserver started at http://127.19.66.193:34867/ using document root <none> and password file <none>
I20260620 19:12:47.361183 22828 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:47.361230 22828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:47.361358 22828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:47.362432 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "d646fc4b78a846a19c8a66575e0c2c43"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "b68fbf5e2ff063f315bb80581c27a265"
server_key_iv: "e53fc75981441e6bb445e18415af5264"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.362773 22828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "d646fc4b78a846a19c8a66575e0c2c43"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "b68fbf5e2ff063f315bb80581c27a265"
server_key_iv: "e53fc75981441e6bb445e18415af5264"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.364039 22828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:12:47.364775 22843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.364944 22828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:47.365027 22828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "d646fc4b78a846a19c8a66575e0c2c43"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "b68fbf5e2ff063f315bb80581c27a265"
server_key_iv: "e53fc75981441e6bb445e18415af5264"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.365118 22828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:47.372732 22828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:47.374475 22828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:47.374607 22828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:47.374810 22828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:47.375106 22828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:47.375138 22828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.375176 22828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:47.375200 22828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.380884 22828 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:42155
I20260620 19:12:47.380939 22956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:42155 every 8 connection(s)
I20260620 19:12:47.381245 22828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:47.389412 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.381868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:44481 (local address 127.19.66.254:33159)
0620 19:12:47.381942 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:47.381944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:47.382104 (+ 160us) server_negotiation.cc:407] Connection header received
0620 19:12:47.382561 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:47.382562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:47.382576 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:47.382621 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:47.383466 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.383960 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.385024 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.385219 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.385270 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.387168 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:47.387169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:47.387169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:47.387176 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:47.388380 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.388681 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.388682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.388682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.388739 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.388949 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.388949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.388950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.389041 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:47.389089 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:47.389287 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:12:47.389347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:12:47.389568 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.381839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33159 (local address 127.19.66.193:44481)
0620 19:12:47.381991 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.382027 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:47.382488 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:47.382682 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:47.382686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:47.383217 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:47.383402 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.383406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.384021 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.384021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.384959 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.384962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.385277 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.385277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.385371 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.385775 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:47.385782 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:47.387093 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:47.388438 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.388441 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.388443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.388632 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.388782 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.388782 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.388783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.388908 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.389090 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:47.389092 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:47.389196 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 19:12:47.389270 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:12:47.389376 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260620 19:12:47.389999 22957 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33159
I20260620 19:12:47.390077 22957 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:47.390215 22957 heartbeater.cc:511] Master 127.19.66.254:33159 requested a full tablet report, sending...
I20260620 19:12:47.390437 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22828
I20260620 19:12:47.390532 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:47.390702 19723 external_mini_cluster.cc:1428] Setting key 9ca5957405da49d93f91aa72360d884f
I20260620 19:12:47.390681 22766 ts_manager.cc:194] Registered new tserver with Master: d646fc4b78a846a19c8a66575e0c2c43 (127.19.66.193:42155)
I20260620 19:12:47.391275 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:44481
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:47.411648 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:47.491408 22962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:47.491621 22962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:47.491647 22962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:47.492771 22962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:47.492811 22962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:47.492837 22962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:47.494304 22962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:47.494527 22962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:47.494745 22962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:47.499547 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.499523 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.499529 22967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:47.499728 22962 server_base.cc:1034] running on GCE node
I20260620 19:12:47.500006 22962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:47.500218 22962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:47.501350 22962 hybrid_clock.cc:648] HybridClock initialized: now 1781982767501335 us; error 32 us; skew 500 ppm
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:47.503787 22962 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:47.504230 22962 webserver.cc:460] Webserver started at http://127.19.66.194:45973/ using document root <none> and password file <none>
I20260620 19:12:47.504452 22962 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:47.504503 22962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:47.504635 22962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:47.505710 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "ce3828bda0874a6a8ba4c870f0e9c202"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "ae10c84b81652d4f2a250e3a50c7a230"
server_key_iv: "708cdd5bbc85e526c1f392694a09c289"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.506067 22962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "ce3828bda0874a6a8ba4c870f0e9c202"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "ae10c84b81652d4f2a250e3a50c7a230"
server_key_iv: "708cdd5bbc85e526c1f392694a09c289"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.507330 22962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:47.508049 22977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.508193 22962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:47.508280 22962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "ce3828bda0874a6a8ba4c870f0e9c202"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "ae10c84b81652d4f2a250e3a50c7a230"
server_key_iv: "708cdd5bbc85e526c1f392694a09c289"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.508348 22962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:47.516585 22962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:47.518477 22962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:47.518654 22962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:47.518877 22962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:47.519176 22962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:47.519208 22962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.519237 22962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:47.519254 22962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.524619 22962 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:37631
I20260620 19:12:47.524667 23090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:37631 every 8 connection(s)
I20260620 19:12:47.525044 22962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:47.526484 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 22962
I20260620 19:12:47.526583 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:47.526778 19723 external_mini_cluster.cc:1428] Setting key 843ae261ab4f0765000f24107aed881a
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
I20260620 19:12:47.535166 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.525737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:42583 (local address 127.19.66.254:33159)
0620 19:12:47.525798 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:47.525800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:47.525974 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:12:47.526505 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:47.526506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:47.526518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:47.526569 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:47.527634 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.528117 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.529520 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.529798 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.529875 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.532346 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:47.532348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:47.532348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:47.532357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:47.533849 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.534263 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.534264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.534265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.534324 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.534554 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.534554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.534555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.534695 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:47.534767 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:47.535018 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:12:47.535097 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:12:47.535346 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.525708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33159 (local address 127.19.66.194:42583)
0620 19:12:47.525857 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.525895 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:47.526402 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:47.526644 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:47.526649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:47.527322 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:47.527551 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.527556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.528195 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.528196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.529427 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.529430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.529912 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.529913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.530086 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.530650 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:47.530658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:47.532246 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:47.533931 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.533936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.533938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.534187 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.534397 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.534398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.534398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.534499 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.534762 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:47.534766 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:47.534895 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:12:47.534978 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:12:47.535133 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 19:12:47.535787 23091 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33159
I20260620 19:12:47.535871 23091 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:47.536046 23091 heartbeater.cc:511] Master 127.19.66.254:33159 requested a full tablet report, sending...
I20260620 19:12:47.536480 22766 ts_manager.cc:194] Registered new tserver with Master: ce3828bda0874a6a8ba4c870f0e9c202 (127.19.66.194:37631)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
I20260620 19:12:47.536921 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:42583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:47.549147 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:47.627365 23096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:47.627571 23096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:47.627601 23096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:47.628748 23096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:47.628789 23096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:47.628821 23096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:47.630278 23096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:47.630522 23096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:47.630725 23096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:47.635440 23101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:47.635558 23096 server_base.cc:1034] running on GCE node
W20260620 19:12:47.635478 23104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:47.635447 23102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:47.635908 23096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:47.636219 23096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:47.637359 23096 hybrid_clock.cc:648] HybridClock initialized: now 1781982767637345 us; error 29 us; skew 500 ppm
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:47.639643 23096 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:47.640089 23096 webserver.cc:460] Webserver started at http://127.19.66.195:43657/ using document root <none> and password file <none>
I20260620 19:12:47.640321 23096 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:47.640369 23096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:47.640491 23096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:47.641539 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "08297dc504b8454790c831f31ea88491"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "2b328ead96140191762c7db6ac9274a3"
server_key_iv: "7b41519f74585f584088930c09999267"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.641858 23096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "08297dc504b8454790c831f31ea88491"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "2b328ead96140191762c7db6ac9274a3"
server_key_iv: "7b41519f74585f584088930c09999267"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.643049 23096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:47.643724 23111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.643872 23096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:47.643951 23096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "08297dc504b8454790c831f31ea88491"
format_stamp: "Formatted at 2026-06-20 19:12:47 on dist-test-slave-nq8r"
server_key: "2b328ead96140191762c7db6ac9274a3"
server_key_iv: "7b41519f74585f584088930c09999267"
server_key_version: "encryptionkey@0"
I20260620 19:12:47.644013 23096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:47.660972 23096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:47.662689 23096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:47.662840 23096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:47.663040 23096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:47.663327 23096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:47.663358 23096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.663386 23096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:47.663403 23096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:47.668731 23096 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:38209
I20260620 19:12:47.668793 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:38209 every 8 connection(s)
I20260620 19:12:47.669094 23096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:47.673501 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23096
I20260620 19:12:47.673599 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:47.673799 19723 external_mini_cluster.cc:1428] Setting key 0118a487bc3e2bbb5c06579c86b85e89
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:47.677281 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.669783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:38215 (local address 127.19.66.254:33159)
0620 19:12:47.669843 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:47.669845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:47.670019 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:12:47.670500 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:47.670501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:47.670510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:47.670565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:47.671440 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.671920 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.672832 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.673020 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.673062 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.675182 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:47.675183 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:47.675183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:47.675190 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:47.676237 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.676551 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.676552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.676552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.676594 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.676763 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.676763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.676763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.676880 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:47.676937 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:47.677133 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:12:47.677213 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:12:47.677441 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.669761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33159 (local address 127.19.66.195:38215)
0620 19:12:47.669904 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:47.669944 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:47.670422 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:47.670627 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:47.670631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:47.671170 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:47.671362 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.671367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.671982 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.671982 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.672774 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:47.672775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.673070 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.673071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:47.673163 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.673608 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:47.673618 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:47.675094 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:47.676297 (+ 1203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.676303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.676304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.676498 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.676636 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:47.676636 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:47.676637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:47.676722 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:47.676927 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:47.676930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:47.677039 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:12:47.677115 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:12:47.677241 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 19:12:47.677846 23225 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33159
I20260620 19:12:47.677928 23225 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:47.678087 23225 heartbeater.cc:511] Master 127.19.66.254:33159 requested a full tablet report, sending...
I20260620 19:12:47.678423 22766 ts_manager.cc:194] Registered new tserver with Master: 08297dc504b8454790c831f31ea88491 (127.19.66.195:38209)
I20260620 19:12:47.678779 22766 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:38215
I20260620 19:12:47.685437 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[22721](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982767, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:47.705405 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:47.699305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.19.66.254:33159)
0620 19:12:47.699380 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:47.699382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:47.699434 (+ 52us) server_negotiation.cc:407] Connection header received
0620 19:12:47.699495 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:47.699496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:47.699504 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:47.699542 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:47.700037 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.700517 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.701246 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:47.701423 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:47.701468 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:47.703403 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:47.703404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:47.703404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:47.703410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:47.704461 (+ 1051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.704747 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.704748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.704748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.704793 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:47.704967 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:47.704968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:47.704968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:47.705113 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:47.705187 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:47.705271 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:12:47.705330 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:12:47.705618 22766 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49356
I20260620 19:12:47.706068 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22828
I20260620 19:12:47.709800 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22962
I20260620 19:12:47.713887 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23096
I20260620 19:12:47.717813 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 22734
2026-06-20T19:12:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1673 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[23237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[23237](info): set up 2 sockets
Jun 20 19:12:47 dist-test-slave-nq8r krb5kdc[23237](info): commencing operation
krb5kdc: starting...
W20260620 19:12:48.652444 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.916s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:48 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:48.756934 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46311
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:48.836172 23250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:48.836365 23250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:48.836385 23250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:48.837538 23250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:48.837579 23250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:48.837592 23250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:48.837603 23250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:48.837616 23250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:48.839078 23250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:48.839291 23250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:48.839483 23250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:48.843870 23255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:48.843876 23258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:48.843876 23256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:48.844029 23250 server_base.cc:1034] running on GCE node
I20260620 19:12:48.844353 23250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:48.844587 23250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:48.845711 23250 hybrid_clock.cc:648] HybridClock initialized: now 1781982768845699 us; error 28 us; skew 500 ppm
Jun 20 19:12:48 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982768, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:48.847991 23250 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:48.848378 23250 webserver.cc:460] Webserver started at http://127.19.66.254:45855/ using document root <none> and password file <none>
I20260620 19:12:48.848560 23250 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:48.848603 23250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:48.848723 23250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:48.849747 23250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "7f94bca99710449eb75a5f1b42758a3b"
format_stamp: "Formatted at 2026-06-20 19:12:48 on dist-test-slave-nq8r"
server_key: "c72213eb1ef77dd9989b76e8b0527a32"
server_key_iv: "1fffa1c380c6464810d5f43d8027e9a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:48.850068 23250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "7f94bca99710449eb75a5f1b42758a3b"
format_stamp: "Formatted at 2026-06-20 19:12:48 on dist-test-slave-nq8r"
server_key: "c72213eb1ef77dd9989b76e8b0527a32"
server_key_iv: "1fffa1c380c6464810d5f43d8027e9a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:48.851229 23250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:48.851857 23265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:48.851989 23250 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:48.852066 23250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "7f94bca99710449eb75a5f1b42758a3b"
format_stamp: "Formatted at 2026-06-20 19:12:48 on dist-test-slave-nq8r"
server_key: "c72213eb1ef77dd9989b76e8b0527a32"
server_key_iv: "1fffa1c380c6464810d5f43d8027e9a9"
server_key_version: "encryptionkey@0"
I20260620 19:12:48.852128 23250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:48.870277 23250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:48.871945 23250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:48.872069 23250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:48.875583 23250 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:46311
I20260620 19:12:48.875619 23317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:46311 every 8 connection(s)
I20260620 19:12:48.876085 23250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:48.876724 23318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:48.878875 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b: Bootstrap starting.
I20260620 19:12:48.879385 23318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:48.879607 23318 log.cc:824] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:48.880194 23318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b: No bootstrap required, opened a new log
I20260620 19:12:48.881004 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23250
I20260620 19:12:48.881106 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:48.881285 19723 external_mini_cluster.cc:1428] Setting key ed0839c134dd57f3b2b15cc29a785018
I20260620 19:12:48.882586 23318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } }
I20260620 19:12:48.882743 23318 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:48.882781 23318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f94bca99710449eb75a5f1b42758a3b, State: Initialized, Role: FOLLOWER
I20260620 19:12:48.882941 23318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } }
I20260620 19:12:48.883023 23318 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:48.883061 23318 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:48.883109 23318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:48.883701 23318 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } }
I20260620 19:12:48.883788 23318 leader_election.cc:302] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f94bca99710449eb75a5f1b42758a3b; no voters:
I20260620 19:12:48.883929 23318 leader_election.cc:288] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:48.883966 23323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:48.884097 23323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [term 1 LEADER]: Becoming Leader. State: Replica: 7f94bca99710449eb75a5f1b42758a3b, State: Running, Role: LEADER
I20260620 19:12:48.884172 23318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:48.884249 23323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } }
I20260620 19:12:48.884588 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f94bca99710449eb75a5f1b42758a3b. Latest consensus state: current_term: 1 leader_uuid: "7f94bca99710449eb75a5f1b42758a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } } }
I20260620 19:12:48.884828 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:48.885227 23332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:48.884763 23325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f94bca99710449eb75a5f1b42758a3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f94bca99710449eb75a5f1b42758a3b" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46311 } } }
I20260620 19:12:48.885744 23332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:48.885818 23325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:48.887283 23332 catalog_manager.cc:1347] Generated new cluster ID: 837336fc80e446a7a57f226052c7bb8b
I20260620 19:12:48.887336 23332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:48 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:48.891351 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:48.881760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.19.66.254:46311)
0620 19:12:48.881927 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:48.881936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:48.881969 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:48.882555 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:48.882560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:48.882605 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:48.882711 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:48.883423 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:48.884631 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:48.885738 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:48.886207 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:48.886304 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:48.888521 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:48.888525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:48.888525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:48.888536 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:48.890005 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:48.890311 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:48.890314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:48.890314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:48.890407 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:48.890578 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:48.890579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:48.890579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:48.890795 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:48.890946 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:48.891049 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 19:12:48.891187 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260620 19:12:48.899049 23332 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:48.899539 23332 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:48.902969 23332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f94bca99710449eb75a5f1b42758a3b: Generated new TSK 0
I20260620 19:12:48.903127 23332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:48.929179 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:49.017711 23344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:49.017906 23344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:49.017928 23344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:49.019049 23344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:49.019088 23344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:49.019114 23344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:49.020578 23344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:49.020792 23344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:49.021023 23344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:49.026486 23352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:49.026533 23350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:49.026739 23344 server_base.cc:1034] running on GCE node
W20260620 19:12:49.026494 23349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:49.026981 23344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:49.027328 23344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:49.028481 23344 hybrid_clock.cc:648] HybridClock initialized: now 1781982769028461 us; error 37 us; skew 500 ppm
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:49.031277 23344 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:49.031785 23344 webserver.cc:460] Webserver started at http://127.19.66.193:44581/ using document root <none> and password file <none>
I20260620 19:12:49.032025 23344 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:49.032083 23344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:49.032219 23344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:49.033278 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "3d14e755ee0c4db0801bcfa79d9b541c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0835ee489034197810d04914b8fe52ba"
server_key_iv: "8f2f001e879044c0d0bf7894a9475c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.033593 23344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "3d14e755ee0c4db0801bcfa79d9b541c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0835ee489034197810d04914b8fe52ba"
server_key_iv: "8f2f001e879044c0d0bf7894a9475c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.034786 23344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 19:12:49.035410 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.035557 23344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.035634 23344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "3d14e755ee0c4db0801bcfa79d9b541c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0835ee489034197810d04914b8fe52ba"
server_key_iv: "8f2f001e879044c0d0bf7894a9475c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.035699 23344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:49.055305 23344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:49.057524 23344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:49.057659 23344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:49.057873 23344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:49.058197 23344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:49.058230 23344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.058259 23344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:49.058282 23344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.063825 23344 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:38177
I20260620 19:12:49.063869 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:38177 every 8 connection(s)
I20260620 19:12:49.064183 23344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:49.072712 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.064896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:38393 (local address 127.19.66.254:46311)
0620 19:12:49.064942 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:49.064944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:49.065164 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:12:49.066044 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:49.066046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:49.066057 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:49.066109 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:49.066706 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.067198 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.068144 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.068343 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.068390 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.070469 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:49.070470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:49.070471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:49.070479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:49.071626 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.071967 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.071968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.071968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.072016 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.072211 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.072212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.072212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.072316 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:49.072362 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:49.072571 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 19:12:49.072643 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:12:49.072902 23475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.064880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46311 (local address 127.19.66.193:38393)
0620 19:12:49.065024 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.065062 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:49.065934 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:49.066176 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:49.066180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:49.066436 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:49.066641 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.066646 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.067268 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.067269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.068079 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.068081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.068402 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.068402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.068509 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.069024 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:49.069032 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:49.070384 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:49.071694 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.071701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.071703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.071911 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.072059 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.072059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.072059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.072149 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.072357 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:49.072361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:49.072477 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 19:12:49.072553 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:12:49.072669 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260620 19:12:49.073390 23473 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46311
I20260620 19:12:49.073479 23473 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:49.073625 23473 heartbeater.cc:511] Master 127.19.66.254:46311 requested a full tablet report, sending...
I20260620 19:12:49.074076 23282 ts_manager.cc:194] Registered new tserver with Master: 3d14e755ee0c4db0801bcfa79d9b541c (127.19.66.193:38177)
I20260620 19:12:49.074282 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23344
I20260620 19:12:49.074358 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:49.074532 19723 external_mini_cluster.cc:1428] Setting key 221fc462ba1e33523afa633e92d47890
I20260620 19:12:49.074851 23282 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:38393
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:49.096354 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:49.181125 23478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:49.181324 23478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:49.181345 23478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:49.182453 23478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:49.182502 23478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:49.182538 23478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:49.183997 23478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:49.184252 23478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:49.184458 23478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:49.188988 23483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:49.189067 23478 server_base.cc:1034] running on GCE node
W20260620 19:12:49.188975 23484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:49.189004 23486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:49.189406 23478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:49.189625 23478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:49.190799 23478 hybrid_clock.cc:648] HybridClock initialized: now 1781982769190789 us; error 25 us; skew 500 ppm
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:49.193095 23478 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:49.193513 23478 webserver.cc:460] Webserver started at http://127.19.66.194:43543/ using document root <none> and password file <none>
I20260620 19:12:49.193707 23478 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:49.193763 23478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:49.193882 23478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:49.194862 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "e36452a3687a416c878311691b41d21c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "29ce33150ec0e2a8a4832dfa64faf77d"
server_key_iv: "ecd38d48ce556365a7a054bde46b711f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.195170 23478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "e36452a3687a416c878311691b41d21c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "29ce33150ec0e2a8a4832dfa64faf77d"
server_key_iv: "ecd38d48ce556365a7a054bde46b711f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.196334 23478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:12:49.196981 23493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.197122 23478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.197198 23478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "e36452a3687a416c878311691b41d21c"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "29ce33150ec0e2a8a4832dfa64faf77d"
server_key_iv: "ecd38d48ce556365a7a054bde46b711f"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.197264 23478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:49.227788 23478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:49.229688 23478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:49.229839 23478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:49.230060 23478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:49.230348 23478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:49.230381 23478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.230409 23478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:49.230432 23478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.235772 23478 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:44071
I20260620 19:12:49.235833 23606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:44071 every 8 connection(s)
I20260620 19:12:49.236140 23478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:49.244499 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.236830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:59617 (local address 127.19.66.254:46311)
0620 19:12:49.236908 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:49.236910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:49.237115 (+ 205us) server_negotiation.cc:407] Connection header received
0620 19:12:49.237977 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:49.237978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:49.237987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:49.238044 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:49.238670 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.239161 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.240138 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.240347 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.240397 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.242271 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:49.242273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:49.242273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:49.242280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:49.243330 (+ 1050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.243672 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.243674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.243674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.243724 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.243923 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.243924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.243924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.244024 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:49.244077 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:49.244324 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 19:12:49.244410 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 19:12:49.244658 23609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.236814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46311 (local address 127.19.66.194:59617)
0620 19:12:49.236988 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.237029 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:49.237886 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:49.238105 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:49.238108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:49.238407 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:49.238600 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.238605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.239248 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.239248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.240071 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.240073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.240406 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.240406 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.240500 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.240937 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:49.240945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:49.242187 (+ 1242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:49.243400 (+ 1213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.243406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.243408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.243610 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.243776 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.243777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.243777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.243870 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.244077 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:49.244083 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:49.244211 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:12:49.244298 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:12:49.244449 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260620 19:12:49.245080 23607 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46311
I20260620 19:12:49.245162 23607 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:49.245303 23607 heartbeater.cc:511] Master 127.19.66.254:46311 requested a full tablet report, sending...
I20260620 19:12:49.245625 23281 ts_manager.cc:194] Registered new tserver with Master: e36452a3687a416c878311691b41d21c (127.19.66.194:44071)
I20260620 19:12:49.245977 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23478
I20260620 19:12:49.245985 23281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:59617
I20260620 19:12:49.246052 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:49.246232 19723 external_mini_cluster.cc:1428] Setting key 03e4193f24eac8828ea907d04ed0dd57
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:49.267279 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:49.345433 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:49.345638 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:49.345660 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:49.346778 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:49.346818 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:49.346846 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:49.348308 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:49.348517 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:49.348724 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:12:49.353363 23612 server_base.cc:1034] running on GCE node
W20260620 19:12:49.353318 23620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:49.353329 23617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:49.353329 23618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:49.353725 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:49.353950 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:49.355170 23612 hybrid_clock.cc:648] HybridClock initialized: now 1781982769355114 us; error 30 us; skew 500 ppm
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:49.357591 23612 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:49.357978 23612 webserver.cc:460] Webserver started at http://127.19.66.195:39125/ using document root <none> and password file <none>
I20260620 19:12:49.358160 23612 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:49.358204 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:49.358337 23612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:49.359319 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "61d32ed521d0484dbc16efb7363ed6e8"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0d57e33cf0a34126277da782856cfd9a"
server_key_iv: "12a8566c25cdb84aef6d49e01e609888"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.359629 23612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "61d32ed521d0484dbc16efb7363ed6e8"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0d57e33cf0a34126277da782856cfd9a"
server_key_iv: "12a8566c25cdb84aef6d49e01e609888"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.360796 23612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:49.361457 23627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.361588 23612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:49.361666 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "61d32ed521d0484dbc16efb7363ed6e8"
format_stamp: "Formatted at 2026-06-20 19:12:49 on dist-test-slave-nq8r"
server_key: "0d57e33cf0a34126277da782856cfd9a"
server_key_iv: "12a8566c25cdb84aef6d49e01e609888"
server_key_version: "encryptionkey@0"
I20260620 19:12:49.361723 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:49.377839 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:49.379554 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:49.379673 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:49.379859 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:49.380151 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:49.380188 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.380218 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:49.380240 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:49.385684 23612 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:43047
I20260620 19:12:49.385769 23740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:43047 every 8 connection(s)
I20260620 19:12:49.386034 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:49.391515 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23612
I20260620 19:12:49.391618 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:49.391793 19723 external_mini_cluster.cc:1428] Setting key 277dc916da896b0c0d578da8af46d7b0
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:49.394766 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.386695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:45801 (local address 127.19.66.254:46311)
0620 19:12:49.386770 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:49.386773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:49.386910 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:12:49.387869 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:49.387870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:49.387881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:49.387943 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:49.388506 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.389071 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.390039 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.390259 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.390311 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.392459 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:49.392460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:49.392460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:49.392467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:49.393744 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.394082 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.394083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.394083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.394129 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.394321 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.394321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.394321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.394414 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:49.394459 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:49.394650 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:12:49.394706 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:12:49.394948 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.386662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46311 (local address 127.19.66.195:45801)
0620 19:12:49.386799 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:49.386833 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:49.387712 (+ 879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:49.387999 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:49.388002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:49.388243 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:49.388426 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.388430 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.389190 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.389191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.389966 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:49.389968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.390318 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.390319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:49.390416 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.390879 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:49.390886 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:49.392393 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:49.393817 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.393821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.393823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.394022 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.394172 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:49.394172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:49.394173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:49.394275 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:49.394457 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:49.394459 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:49.394564 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 19:12:49.394638 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:12:49.394746 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260620 19:12:49.395326 23741 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46311
I20260620 19:12:49.395409 23741 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:49.395567 23741 heartbeater.cc:511] Master 127.19.66.254:46311 requested a full tablet report, sending...
I20260620 19:12:49.395865 23281 ts_manager.cc:194] Registered new tserver with Master: 61d32ed521d0484dbc16efb7363ed6e8 (127.19.66.195:43047)
I20260620 19:12:49.396198 23281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:45801
I20260620 19:12:49.403348 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:49.424671 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.417885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.19.66.193:38177)
0620 19:12:49.418028 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:49.418034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:49.418046 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:49.418073 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:49.418075 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:49.418088 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:49.418158 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:49.418674 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.419386 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.420068 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.420385 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.420433 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.422463 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:49.422474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:49.422475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:49.422482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:49.423759 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.424057 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.424059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.424059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.424105 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.424248 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.424248 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.424248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.424382 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:49.424483 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:49.424534 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:12:49.424595 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:49.431761 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:49.425905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.19.66.254:46311)
0620 19:12:49.425955 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:49.425956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:49.425965 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:49.425985 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:49.425986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:49.425994 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:49.426035 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:49.426414 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.426894 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.427549 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:49.427761 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:49.427823 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:49.429754 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:49.429756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:49.429756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:49.429763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:49.430960 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.431201 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.431202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.431202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.431247 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:49.431364 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:49.431365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:49.431365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:49.431490 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:49.431572 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:49.431621 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:12:49.431695 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 19:12:49.431972 23281 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60304
I20260620 19:12:49.432263 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23344
I20260620 19:12:49.436323 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23478
I20260620 19:12:49.439962 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23612
I20260620 19:12:49.443483 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23250
2026-06-20T19:12:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1725 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23753](info): set up 2 sockets
Jun 20 19:12:49 dist-test-slave-nq8r krb5kdc[23753](info): commencing operation
krb5kdc: starting...
W20260620 19:12:50.388548 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:50.495518 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35117
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:50.572757 23766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:50.572970 23766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:50.572990 23766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:50.574086 23766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:50.574116 23766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:50.574128 23766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:50.574138 23766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:50.574149 23766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:50.575554 23766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:50.575734 23766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:50.575903 23766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:50.580240 23771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.580242 23774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.580329 23772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:50.580222 23766 server_base.cc:1034] running on GCE node
I20260620 19:12:50.580648 23766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:50.580945 23766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:50.582082 23766 hybrid_clock.cc:648] HybridClock initialized: now 1781982770582067 us; error 27 us; skew 500 ppm
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:50.584388 23766 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:50.584728 23766 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:50.584779 23766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:50.584926 23766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:50.585980 23766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "316df00b54dd4ff1b9ce5ca415425938"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5b7a14226eb440bdbe402234fbeb7845"
server_key_iv: "5c3ec341f1ac125390dd533ffb82b931"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.586319 23766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "316df00b54dd4ff1b9ce5ca415425938"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5b7a14226eb440bdbe402234fbeb7845"
server_key_iv: "5c3ec341f1ac125390dd533ffb82b931"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.587458 23766 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:50.588054 23780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:50.588219 23766 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:50.588297 23766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "316df00b54dd4ff1b9ce5ca415425938"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5b7a14226eb440bdbe402234fbeb7845"
server_key_iv: "5c3ec341f1ac125390dd533ffb82b931"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.588362 23766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:50.611534 23766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:50.613302 23766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:50.613428 23766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:50.616986 23766 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:35117
I20260620 19:12:50.617030 23832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:35117 every 8 connection(s)
I20260620 19:12:50.617311 23766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:50.618157 23833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:50.619931 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23766
I20260620 19:12:50.620015 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:50.620172 19723 external_mini_cluster.cc:1428] Setting key 71503e08449e6a97946a081ed1c1526f
I20260620 19:12:50.620520 23833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: Bootstrap starting.
I20260620 19:12:50.621198 23833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:50.621503 23833 log.cc:824] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:50.622165 23833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: No bootstrap required, opened a new log
I20260620 19:12:50.624257 23833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } }
I20260620 19:12:50.624404 23833 raft_consensus.cc:374] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:50.624449 23833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316df00b54dd4ff1b9ce5ca415425938, State: Initialized, Role: FOLLOWER
I20260620 19:12:50.624560 23833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } }
I20260620 19:12:50.624634 23833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:50.624666 23833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:50.624708 23833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:50.625350 23833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } }
I20260620 19:12:50.625454 23833 leader_election.cc:302] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 316df00b54dd4ff1b9ce5ca415425938; no voters:
I20260620 19:12:50.625622 23833 leader_election.cc:288] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:50.625660 23838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:50.625792 23838 raft_consensus.cc:686] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [term 1 LEADER]: Becoming Leader. State: Replica: 316df00b54dd4ff1b9ce5ca415425938, State: Running, Role: LEADER
I20260620 19:12:50.625862 23833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:50.625901 23838 consensus_queue.cc:237] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } }
I20260620 19:12:50.626231 23838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 316df00b54dd4ff1b9ce5ca415425938. Latest consensus state: current_term: 1 leader_uuid: "316df00b54dd4ff1b9ce5ca415425938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } } }
I20260620 19:12:50.626309 23838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:50.626240 23840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "316df00b54dd4ff1b9ce5ca415425938" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "316df00b54dd4ff1b9ce5ca415425938" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35117 } } }
I20260620 19:12:50.626457 23840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
W20260620 19:12:50.627465 23853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:50.627517 23853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:50.628089 23845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:50.628242 23845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:50.629491 23845 catalog_manager.cc:1347] Generated new cluster ID: 388be60276af45149d3e62342181e0b0
I20260620 19:12:50.629537 23845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:50.630993 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.620626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50518 (local address 127.19.66.254:35117)
0620 19:12:50.620806 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:50.620816 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:50.620873 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:12:50.621440 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:50.621449 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:50.621529 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:50.621674 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:50.622281 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.623617 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.624355 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.624962 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.625067 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:50.627545 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:50.627548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:50.627549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:50.627560 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:50.629477 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:50.629853 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:50.629856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:50.629856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:50.629950 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:50.630164 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:50.630165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:50.630165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:50.630429 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:50.630563 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:50.630675 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:12:50.630796 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260620 19:12:50.658375 23845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:50.659135 23845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:50.678190 23845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: Generated new TSK 0
I20260620 19:12:50.678407 23845 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:50.707577 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:50.809181 23859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:50.809366 23859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:50.809388 23859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:50.810467 23859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:50.810498 23859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:50.810523 23859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:50.811926 23859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:50.812100 23859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:50.812270 23859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:50.816735 23865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.816751 23864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.816735 23867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:50.816934 23859 server_base.cc:1034] running on GCE node
I20260620 19:12:50.817170 23859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:50.817364 23859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:50.818493 23859 hybrid_clock.cc:648] HybridClock initialized: now 1781982770818482 us; error 28 us; skew 500 ppm
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:50.820753 23859 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:50.821075 23859 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:50.821121 23859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:50.821237 23859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:50.822266 23859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "371e2e680ff34327b1f50240a6e32d07"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5d33af16fe8ad7f8c40372e4adccd4ab"
server_key_iv: "5c9757c44b917901d9d3ac202691e559"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.822574 23859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "371e2e680ff34327b1f50240a6e32d07"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5d33af16fe8ad7f8c40372e4adccd4ab"
server_key_iv: "5c9757c44b917901d9d3ac202691e559"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.823746 23859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:12:50.824373 23873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:50.824503 23859 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:50.824575 23859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "371e2e680ff34327b1f50240a6e32d07"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "5d33af16fe8ad7f8c40372e4adccd4ab"
server_key_iv: "5c9757c44b917901d9d3ac202691e559"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.824635 23859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:50.841662 23859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:50.843822 23859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:50.843954 23859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:50.844169 23859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:50.844460 23859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:50.844492 23859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:50.844521 23859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:50.844544 23859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:50.849762 23859 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:44381
I20260620 19:12:50.849807 23986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:44381 every 8 connection(s)
I20260620 19:12:50.850049 23859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:50.852370 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23859
I20260620 19:12:50.852463 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:50.852627 19723 external_mini_cluster.cc:1428] Setting key 7719853cd4a0fdd2ee2958ce87e6fe81
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:50.859438 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.850695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:40739 (local address 127.19.66.254:35117)
0620 19:12:50.850770 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:50.850773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:50.850934 (+ 161us) server_negotiation.cc:407] Connection header received
0620 19:12:50.851373 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:50.851374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:50.851383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:50.851432 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:50.852592 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.853345 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.854322 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.854534 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.854589 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:50.856813 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:50.856814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:50.856815 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:50.856822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:50.858223 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:50.858572 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:50.858573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:50.858574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:50.858633 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:50.858825 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:50.858826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:50.858826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:50.858959 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:50.859028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:50.859268 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:12:50.859349 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:12:50.859477 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:50.850672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35117 (local address 127.19.66.193:40739)
0620 19:12:50.850825 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:50.850859 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:50.851301 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:50.851486 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:50.851489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:50.851801 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:50.852467 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:50.852475 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.853444 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.853444 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:50.854242 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:50.854245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:50.854599 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:50.854600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:50.854726 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:50.855166 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:50.855174 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:50.856710 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:50.858303 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:50.858309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:50.858311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:50.858507 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:50.858690 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:50.858690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:50.858690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:50.858775 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:50.859013 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:50.859016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:50.859122 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:12:50.859194 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:12:50.859320 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260620 19:12:50.860030 23987 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35117
I20260620 19:12:50.860127 23987 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:50.860280 23987 heartbeater.cc:511] Master 127.19.66.254:35117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
I20260620 19:12:50.860766 23797 ts_manager.cc:194] Registered new tserver with Master: 371e2e680ff34327b1f50240a6e32d07 (127.19.66.193:44381)
I20260620 19:12:50.861438 23797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:40739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:50.875085 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:50.952180 23992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:50.952368 23992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:50.952389 23992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:50.953538 23992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:50.953579 23992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:50.953604 23992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:50.955099 23992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:50.955327 23992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:50.955535 23992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:50.960007 23998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.960028 24000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:50.960013 23997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:50.960255 23992 server_base.cc:1034] running on GCE node
I20260620 19:12:50.960466 23992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:50.960690 23992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:50.961831 23992 hybrid_clock.cc:648] HybridClock initialized: now 1781982770961815 us; error 27 us; skew 500 ppm
Jun 20 19:12:50 dist-test-slave-nq8r krb5kdc[23753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:50.963994 23992 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:50.964309 23992 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:50.964360 23992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:50.964483 23992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:50.965510 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "2c2c78bb5b6f4bba8635f9869d046437"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "04cdfb3cef098d117328344bbe3a14bf"
server_key_iv: "b37f5b31816229ee67e1d00d275cbcc0"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.965826 23992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "2c2c78bb5b6f4bba8635f9869d046437"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "04cdfb3cef098d117328344bbe3a14bf"
server_key_iv: "b37f5b31816229ee67e1d00d275cbcc0"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.966962 23992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:50.967572 24006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:50.967728 23992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:50.967813 23992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "2c2c78bb5b6f4bba8635f9869d046437"
format_stamp: "Formatted at 2026-06-20 19:12:50 on dist-test-slave-nq8r"
server_key: "04cdfb3cef098d117328344bbe3a14bf"
server_key_iv: "b37f5b31816229ee67e1d00d275cbcc0"
server_key_version: "encryptionkey@0"
I20260620 19:12:50.967877 23992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.002293 23992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.004109 23992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.004248 23992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.004467 23992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:51.004783 23992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:51.004817 23992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.004866 23992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:51.004895 23992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.010195 23992 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:42347
I20260620 19:12:51.010249 24119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:42347 every 8 connection(s)
I20260620 19:12:51.010488 23992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:12:51 dist-test-slave-nq8r krb5kdc[23753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982770, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:51.018776 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.011086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:46497 (local address 127.19.66.254:35117)
0620 19:12:51.011167 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.011169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.011334 (+ 165us) server_negotiation.cc:407] Connection header received
0620 19:12:51.011771 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.011772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.011782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.011827 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.012736 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.013301 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.014251 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.014431 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.014482 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.016372 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.016374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.016374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:51.016381 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.017536 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.017948 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.017949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.017950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.018001 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.018212 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.018213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.018214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.018313 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.018367 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.018613 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.018689 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:12:51.018939 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.011061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35117 (local address 127.19.66.194:46497)
0620 19:12:51.011216 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.011251 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.011699 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.011891 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.011895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.012143 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.012640 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.012646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.013371 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.013372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.014162 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.014163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.014489 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.014490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.014583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.014977 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:51.014984 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.016278 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.017607 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:51.017613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:51.017615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:51.017884 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:51.018052 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:51.018052 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:51.018053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:51.018155 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:51.018365 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.018372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.018499 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.018585 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.018737 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260620 19:12:51.019340 24120 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35117
I20260620 19:12:51.019388 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 23992
I20260620 19:12:51.019423 24120 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:51.019486 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:51.019553 24120 heartbeater.cc:511] Master 127.19.66.254:35117 requested a full tablet report, sending...
I20260620 19:12:51.019655 19723 external_mini_cluster.cc:1428] Setting key 2ee7d116c523a73b59021e6194103e95
I20260620 19:12:51.019898 23797 ts_manager.cc:194] Registered new tserver with Master: 2c2c78bb5b6f4bba8635f9869d046437 (127.19.66.194:42347)
I20260620 19:12:51.020442 23797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:46497
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:51.040753 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:37069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:51.116830 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:51.117049 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:51.117079 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:51.118235 24125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:51.118276 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:51.118320 24125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:51.119776 24125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:51.119992 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:51.120185 24125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:51.124715 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.124725 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.124699 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.124961 24125 server_base.cc:1034] running on GCE node
I20260620 19:12:51.125166 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:51.125375 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:51.126506 24125 hybrid_clock.cc:648] HybridClock initialized: now 1781982771126484 us; error 38 us; skew 500 ppm
Jun 20 19:12:51 dist-test-slave-nq8r krb5kdc[23753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982771, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:51.128993 24125 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:51.129290 24125 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:51.129335 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:51.129455 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:51.130437 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "39f99f45eced47c4827f8b45551a0726"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "ce93788d65fd7c3a18f7af9b388954e7"
server_key_iv: "009cfe9e1397afb3fb247d8d2e2755f3"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.130752 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "39f99f45eced47c4827f8b45551a0726"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "ce93788d65fd7c3a18f7af9b388954e7"
server_key_iv: "009cfe9e1397afb3fb247d8d2e2755f3"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.131872 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:51.132450 24139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.132587 24125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:51.132670 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "39f99f45eced47c4827f8b45551a0726"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "ce93788d65fd7c3a18f7af9b388954e7"
server_key_iv: "009cfe9e1397afb3fb247d8d2e2755f3"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.132721 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.141004 24125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.142894 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.142992 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.143148 24125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:51.143379 24125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:51.143407 24125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.143432 24125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:51.143448 24125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.148494 24125 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:42411
I20260620 19:12:51.148555 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:42411 every 8 connection(s)
I20260620 19:12:51.148746 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:12:51 dist-test-slave-nq8r krb5kdc[23753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982771, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:51.154443 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24125
I20260620 19:12:51.154512 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:51.154670 19723 external_mini_cluster.cc:1428] Setting key e4b952a74fd7561032dd85b112a37ecd
I20260620 19:12:51.156952 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.149382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:52569 (local address 127.19.66.254:35117)
0620 19:12:51.149428 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.149430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.149602 (+ 172us) server_negotiation.cc:407] Connection header received
0620 19:12:51.150070 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.150071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.150084 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.150130 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.151013 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.151511 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.152463 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.152657 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.152719 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.154543 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.154545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.154545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:51.154553 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.155887 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.156217 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.156217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.156218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.156261 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.156419 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.156419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.156420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.156509 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.156555 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.156756 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.156865 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:12:51.157045 24255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.149356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35117 (local address 127.19.66.195:52569)
0620 19:12:51.149483 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.149522 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.149973 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.150216 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.150220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.150457 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.150900 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.150905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.151588 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.151589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.152378 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.152380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.152721 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.152722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.152839 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.153206 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:51.153233 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.154454 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.155949 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:51.155954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:51.155956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:51.156164 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:51.156297 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:51.156298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:51.156298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:51.156379 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:51.156552 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.156554 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.156657 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.156736 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.156841 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260620 19:12:51.157420 24253 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35117
I20260620 19:12:51.157498 24253 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:51.157627 24253 heartbeater.cc:511] Master 127.19.66.254:35117 requested a full tablet report, sending...
I20260620 19:12:51.157920 23797 ts_manager.cc:194] Registered new tserver with Master: 39f99f45eced47c4827f8b45551a0726 (127.19.66.195:42411)
I20260620 19:12:51.158267 23797 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:52569
I20260620 19:12:51.166083 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:51.172204 23836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.167208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.19.66.254:35117)
0620 19:12:51.167255 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.167257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.167335 (+ 78us) server_negotiation.cc:407] Connection header received
0620 19:12:51.167383 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.167384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.167391 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.167436 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.167907 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.168369 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.169075 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.169249 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.169293 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.170334 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.170335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.170336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:51.170342 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.171396 (+ 1054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.171671 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.171672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.171672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.171716 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:51.171830 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:51.171831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:51.171831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:51.171953 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.172023 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.172090 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.172144 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:12:51.173547 23797 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:51.175082 23797 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:51.181005 24262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44381 (local address 127.0.0.1:32806)
0620 19:12:51.177469 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.177479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.177529 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.177706 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.177706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.177708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:51.177753 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.177755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.178792 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.178793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.180005 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.180007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.180699 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.180784 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.180797 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.180879 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.180935 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":61,"threads_started":1}
I20260620 19:12:51.181022 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.19.66.193:44381)
0620 19:12:51.177578 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.177582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.177591 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:51.177625 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.177626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.177638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.177701 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:51.177836 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.178700 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180087 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.180627 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180689 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.180750 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.180908 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.180948 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":90,"threads_started":1}
I20260620 19:12:51.181154 24261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:42411 (local address 127.0.0.1:51638)
0620 19:12:51.177176 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.177223 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.177281 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.177676 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.177679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.177682 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:51.177746 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.177747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.179025 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.179026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.180078 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.180080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.180945 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.181005 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.181009 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.181038 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.181077 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260620 19:12:51.181459 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.19.66.195:42411)
0620 19:12:51.177515 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.177520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.177532 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:12:51.177569 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.177571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.177586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.177660 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:51.177797 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.178933 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180134 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.180825 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.181261 (+ 436us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.181331 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.181354 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.181393 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260620 19:12:51.181768 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42347 (local address 127.0.0.1:54728)
0620 19:12:51.177796 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.177826 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.177873 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.178112 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.178113 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.178114 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:51.178154 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.178155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.179275 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.179276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.180135 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.180137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.181022 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.181023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.181621 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.181625 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.181678 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.181717 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":2,"thread_start_us":42,"threads_started":1}
I20260620 19:12:51.181730 23921 tablet_service.cc:1467] Processing CreateTablet for tablet 668c90c269c941d580f40775ccf7aecb (DEFAULT_TABLE table=test-table [id=0320500674c44f50ab7a1ade9002a973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.182040 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.177391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.19.66.194:42347)
0620 19:12:51.177550 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.177553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.177923 (+ 370us) server_negotiation.cc:407] Connection header received
0620 19:12:51.177960 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.177963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.177976 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.178038 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:51.178225 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.179196 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180209 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.180887 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.180942 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.180993 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.181920 (+ 927us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.181963 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260620 19:12:51.182086 23921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c90c269c941d580f40775ccf7aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.182037 24187 tablet_service.cc:1467] Processing CreateTablet for tablet 668c90c269c941d580f40775ccf7aecb (DEFAULT_TABLE table=test-table [id=0320500674c44f50ab7a1ade9002a973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.182369 24187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c90c269c941d580f40775ccf7aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.182556 24054 tablet_service.cc:1467] Processing CreateTablet for tablet 668c90c269c941d580f40775ccf7aecb (DEFAULT_TABLE table=test-table [id=0320500674c44f50ab7a1ade9002a973]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.182821 24054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 668c90c269c941d580f40775ccf7aecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.185014 24267 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Bootstrap starting.
I20260620 19:12:51.185408 24269 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Bootstrap starting.
I20260620 19:12:51.185870 24267 tablet_bootstrap.cc:654] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.185985 24269 tablet_bootstrap.cc:654] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.186462 24267 log.cc:824] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.186498 24269 log.cc:824] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.186771 24268 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Bootstrap starting.
I20260620 19:12:51.187170 24269 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: No bootstrap required, opened a new log
I20260620 19:12:51.187237 24269 ts_tablet_manager.cc:1397] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:51.187273 24267 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: No bootstrap required, opened a new log
I20260620 19:12:51.187340 24267 ts_tablet_manager.cc:1397] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:51.187341 24268 tablet_bootstrap.cc:654] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.187873 24268 log.cc:824] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.188560 24268 tablet_bootstrap.cc:492] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: No bootstrap required, opened a new log
I20260620 19:12:51.188647 24268 ts_tablet_manager.cc:1397] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:51.189010 24269 raft_consensus.cc:348] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.189134 24269 raft_consensus.cc:374] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.189028 24267 raft_consensus.cc:348] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.189165 24269 raft_consensus.cc:729] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371e2e680ff34327b1f50240a6e32d07, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.189195 24267 raft_consensus.cc:374] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.189226 24267 raft_consensus.cc:729] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2c78bb5b6f4bba8635f9869d046437, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.189291 24269 consensus_queue.cc:260] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.189343 24267 consensus_queue.cc:260] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.189548 24269 ts_tablet_manager.cc:1428] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:12:51.189664 23987 heartbeater.cc:503] Master 127.19.66.254:35117 was elected leader, sending a full tablet report...
I20260620 19:12:51.189610 24120 heartbeater.cc:503] Master 127.19.66.254:35117 was elected leader, sending a full tablet report...
I20260620 19:12:51.189596 24267 ts_tablet_manager.cc:1428] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:12:51.190442 24268 raft_consensus.cc:348] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.190582 24268 raft_consensus.cc:374] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.190629 24268 raft_consensus.cc:729] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f99f45eced47c4827f8b45551a0726, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.190750 24268 consensus_queue.cc:260] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.190979 24268 ts_tablet_manager.cc:1428] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 19:12:51.190999 24253 heartbeater.cc:503] Master 127.19.66.254:35117 was elected leader, sending a full tablet report...
W20260620 19:12:51.260995 24121 tablet.cc:2369] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:51.328778 24274 raft_consensus.cc:482] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:51.328928 24274 raft_consensus.cc:504] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.329244 24274 leader_election.cc:288] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f99f45eced47c4827f8b45551a0726 (127.19.66.195:42411), 371e2e680ff34327b1f50240a6e32d07 (127.19.66.193:44381)
I20260620 19:12:51.331897 24263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.329573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50335 (local address 127.19.66.193:44381)
0620 19:12:51.329643 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.329645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.329654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:51.329686 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.329687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.329695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.329749 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:51.329878 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.330379 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331244 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.331596 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331663 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.331671 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.331793 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.331826 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:12:51.331902 24265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.329493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:47603 (local address 127.19.66.195:42411)
0620 19:12:51.329554 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.329556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.329566 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:12:51.329594 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.329595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.329607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.329667 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:12:51.329833 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.330304 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331244 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.331595 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331653 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.331662 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.331799 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.331840 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:12:51.331907 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.329414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:42411 (local address 127.19.66.194:47603)
0620 19:12:51.329463 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.329475 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.329519 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.329680 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.329681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.329683 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:51.329756 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.329758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.330377 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.330378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.331164 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.331166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.331666 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.331740 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.331743 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.331788 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.331826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":28}
I20260620 19:12:51.331908 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.329422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44381 (local address 127.19.66.194:50335)
0620 19:12:51.329594 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.329602 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.329646 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.329755 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.329756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.329757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:12:51.329817 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.329819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.330449 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.330450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.331166 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:51.331168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.331668 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.331669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:51.331734 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.331738 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.331781 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.331819 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":11,"thread_start_us":83,"threads_started":1}
I20260620 19:12:51.332284 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c90c269c941d580f40775ccf7aecb" candidate_uuid: "2c2c78bb5b6f4bba8635f9869d046437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f99f45eced47c4827f8b45551a0726" is_pre_election: true
I20260620 19:12:51.332432 24207 raft_consensus.cc:2454] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2c78bb5b6f4bba8635f9869d046437 in term 0.
I20260620 19:12:51.332393 23941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c90c269c941d580f40775ccf7aecb" candidate_uuid: "2c2c78bb5b6f4bba8635f9869d046437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e2e680ff34327b1f50240a6e32d07" is_pre_election: true
I20260620 19:12:51.332547 23941 raft_consensus.cc:2454] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2c78bb5b6f4bba8635f9869d046437 in term 0.
I20260620 19:12:51.332690 24010 leader_election.cc:302] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2c78bb5b6f4bba8635f9869d046437, 39f99f45eced47c4827f8b45551a0726; no voters:
I20260620 19:12:51.332823 24274 raft_consensus.cc:2781] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:51.332908 24274 raft_consensus.cc:482] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:51.332931 24274 raft_consensus.cc:3037] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:51.333559 24274 raft_consensus.cc:504] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.333668 24274 leader_election.cc:288] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [CANDIDATE]: Term 1 election: Requested vote from peers 39f99f45eced47c4827f8b45551a0726 (127.19.66.195:42411), 371e2e680ff34327b1f50240a6e32d07 (127.19.66.193:44381)
I20260620 19:12:51.333810 24207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c90c269c941d580f40775ccf7aecb" candidate_uuid: "2c2c78bb5b6f4bba8635f9869d046437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f99f45eced47c4827f8b45551a0726"
I20260620 19:12:51.333825 23941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "668c90c269c941d580f40775ccf7aecb" candidate_uuid: "2c2c78bb5b6f4bba8635f9869d046437" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "371e2e680ff34327b1f50240a6e32d07"
I20260620 19:12:51.333884 24207 raft_consensus.cc:3037] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:51.333889 23941 raft_consensus.cc:3037] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:51.334532 23941 raft_consensus.cc:2454] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2c78bb5b6f4bba8635f9869d046437 in term 1.
I20260620 19:12:51.334569 24207 raft_consensus.cc:2454] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2c78bb5b6f4bba8635f9869d046437 in term 1.
I20260620 19:12:51.334703 24008 leader_election.cc:302] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2c78bb5b6f4bba8635f9869d046437, 371e2e680ff34327b1f50240a6e32d07; no voters:
I20260620 19:12:51.334795 24274 raft_consensus.cc:2781] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:51.334949 24274 raft_consensus.cc:686] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 1 LEADER]: Becoming Leader. State: Replica: 2c2c78bb5b6f4bba8635f9869d046437, State: Running, Role: LEADER
I20260620 19:12:51.335032 24274 consensus_queue.cc:237] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } }
I20260620 19:12:51.335772 23796 catalog_manager.cc:5495] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c2c78bb5b6f4bba8635f9869d046437 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "2c2c78bb5b6f4bba8635f9869d046437" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2c78bb5b6f4bba8635f9869d046437" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42347 } health_report { overall_health: HEALTHY } } }
W20260620 19:12:51.350695 23988 tablet.cc:2369] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:51.362957 24266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.360910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54736 (local address 127.19.66.194:42347)
0620 19:12:51.360957 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.360959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.360966 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:51.361000 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.361000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.361017 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.361074 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:12:51.361174 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.361645 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.362355 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:51.362525 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:51.362567 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:51.362688 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:12:51.362780 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:12:51.362836 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.362851 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.362867 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:12:51.364604 24207 raft_consensus.cc:1261] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 1 FOLLOWER]: Refusing update from remote peer 2c2c78bb5b6f4bba8635f9869d046437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:51.364603 23941 raft_consensus.cc:1261] T 668c90c269c941d580f40775ccf7aecb P 371e2e680ff34327b1f50240a6e32d07 [term 1 FOLLOWER]: Refusing update from remote peer 2c2c78bb5b6f4bba8635f9869d046437: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:51.364943 24277 consensus_queue.cc:1035] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "371e2e680ff34327b1f50240a6e32d07" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:51.365063 24274 consensus_queue.cc:1035] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f99f45eced47c4827f8b45551a0726" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:51.367609 24281 mvcc.cc:205] Tried to move back new op lower bound from 7299001431506976768 to 7299001431388758016. Current Snapshot: MvccSnapshot[applied={T|T < 7299001431506976768}]
I20260620 19:12:51.367739 24282 mvcc.cc:205] Tried to move back new op lower bound from 7299001431506976768 to 7299001431388758016. Current Snapshot: MvccSnapshot[applied={T|T < 7299001431506976768}]
I20260620 19:12:51.369328 23796 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:51.369421 23796 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:51.370100 23796 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 316df00b54dd4ff1b9ce5ca415425938: Sending DeleteTablet for 3 replicas of tablet 668c90c269c941d580f40775ccf7aecb
I20260620 19:12:51.370433 23921 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c90c269c941d580f40775ccf7aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:51 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:32806
I20260620 19:12:51.370492 24187 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c90c269c941d580f40775ccf7aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:51 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:51638
I20260620 19:12:51.370639 24054 tablet_service.cc:1514] Processing DeleteTablet for tablet 668c90c269c941d580f40775ccf7aecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:51 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:54728
I20260620 19:12:51.370687 24291 tablet_replica.cc:330] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: stopping tablet replica
I20260620 19:12:51.370810 24291 raft_consensus.cc:2229] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:51.370908 24291 raft_consensus.cc:2258] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:51.371063 24292 tablet_replica.cc:330] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: stopping tablet replica
I20260620 19:12:51.371093 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23859
I20260620 19:12:51.371147 24292 raft_consensus.cc:2229] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:51.371662 24292 raft_consensus.cc:2258] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:51.372098 24291 ts_tablet_manager.cc:1905] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:51.372162 24292 ts_tablet_manager.cc:1905] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:51.374207 24292 ts_tablet_manager.cc:1918] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:51.374279 24292 log.cc:1196] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/668c90c269c941d580f40775ccf7aecb
I20260620 19:12:51.374522 24292 ts_tablet_manager.cc:1939] T 668c90c269c941d580f40775ccf7aecb P 2c2c78bb5b6f4bba8635f9869d046437: Deleting consensus metadata
I20260620 19:12:51.374910 23782 catalog_manager.cc:4841] TS 2c2c78bb5b6f4bba8635f9869d046437 (127.19.66.194:42347): tablet 668c90c269c941d580f40775ccf7aecb (table test-table [id=0320500674c44f50ab7a1ade9002a973]) successfully deleted
I20260620 19:12:51.374944 24291 ts_tablet_manager.cc:1918] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:51.375069 24291 log.cc:1196] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/668c90c269c941d580f40775ccf7aecb
I20260620 19:12:51.375294 24291 ts_tablet_manager.cc:1939] T 668c90c269c941d580f40775ccf7aecb P 39f99f45eced47c4827f8b45551a0726: Deleting consensus metadata
I20260620 19:12:51.375644 23784 catalog_manager.cc:4841] TS 39f99f45eced47c4827f8b45551a0726 (127.19.66.195:42411): tablet 668c90c269c941d580f40775ccf7aecb (table test-table [id=0320500674c44f50ab7a1ade9002a973]) successfully deleted
W20260620 19:12:51.378804 23782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:44381 (error 108)
I20260620 19:12:51.379088 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23992
I20260620 19:12:51.379264 24264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.379123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44381 (local address 127.0.0.1:32814)
0620 19:12:51.379198 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.379233 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:44381: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 19:12:51.379745 23782 catalog_manager.cc:4584] TS 371e2e680ff34327b1f50240a6e32d07 (127.19.66.193:44381): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 668c90c269c941d580f40775ccf7aecb: Network error: Client connection negotiation failed: client connection to 127.19.66.193:44381: connect: Connection refused (error 111)
I20260620 19:12:51.383883 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24125
I20260620 19:12:51.387418 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 23766
2026-06-20T19:12:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1943 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T19:12:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:51Z Disabled control of system clock
I20260620 19:12:51.412115 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37413
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:37413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:12:51.488991 24297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:51.489180 24297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:51.489200 24297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:51.490296 24297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:51.490329 24297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:51.490339 24297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:51.490350 24297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:51.490361 24297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:51.491637 24297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:37413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:51.491830 24297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:51.492053 24297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:51.496426 24302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.496490 24305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.496460 24297 server_base.cc:1034] running on GCE node
W20260620 19:12:51.496569 24303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.496948 24297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:51.497236 24297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:51.498373 24297 hybrid_clock.cc:648] HybridClock initialized: now 1781982771498360 us; error 29 us; skew 500 ppm
I20260620 19:12:51.499519 24297 webserver.cc:460] Webserver started at http://127.19.66.254:36279/ using document root <none> and password file <none>
I20260620 19:12:51.499857 24297 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:51.499914 24297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:51.500038 24297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:51.501092 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "3dccd087fde7407ba6f27934f21a6b39"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "80015b30cd2d27207c417d1daf95f8e8"
server_key_iv: "9528ec276e95f029b2c0f5a7bb7f92a0"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.501417 24297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "3dccd087fde7407ba6f27934f21a6b39"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "80015b30cd2d27207c417d1daf95f8e8"
server_key_iv: "9528ec276e95f029b2c0f5a7bb7f92a0"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.502669 24297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:51.503297 24311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.503436 24297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:51.503515 24297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "3dccd087fde7407ba6f27934f21a6b39"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "80015b30cd2d27207c417d1daf95f8e8"
server_key_iv: "9528ec276e95f029b2c0f5a7bb7f92a0"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.503574 24297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.506772 24297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.506999 24297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.507110 24297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.510596 24297 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:37413
I20260620 19:12:51.510623 24363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:37413 every 8 connection(s)
I20260620 19:12:51.510911 24297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:51.511508 24364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.514118 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: Bootstrap starting.
I20260620 19:12:51.514637 24364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.514892 24364 log.cc:824] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.515476 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: No bootstrap required, opened a new log
I20260620 19:12:51.515719 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24297
I20260620 19:12:51.515789 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:51.515931 19723 external_mini_cluster.cc:1428] Setting key aa2b711ae7070d0a566b573785bfd2c2
I20260620 19:12:51.517431 24364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } }
I20260620 19:12:51.517589 24364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.517623 24364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dccd087fde7407ba6f27934f21a6b39, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.517760 24364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } }
I20260620 19:12:51.517848 24364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:51.517884 24364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:51.517920 24364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:51.518505 24364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } }
I20260620 19:12:51.518612 24364 leader_election.cc:302] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dccd087fde7407ba6f27934f21a6b39; no voters:
I20260620 19:12:51.518651 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.516396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49376 (local address 127.19.66.254:37413)
0620 19:12:51.516588 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.516597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.516631 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:12:51.517400 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.517409 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.517457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.517599 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.518212 (+ 613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.518216 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.518216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.518220 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.518263 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.518278 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.518349 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.518384 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.518436 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260620 19:12:51.518766 24364 leader_election.cc:288] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:51.518985 24364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:51.519158 24370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:51.519289 24370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [term 1 LEADER]: Becoming Leader. State: Replica: 3dccd087fde7407ba6f27934f21a6b39, State: Running, Role: LEADER
I20260620 19:12:51.519368 24370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } }
I20260620 19:12:51.519697 24370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dccd087fde7407ba6f27934f21a6b39. Latest consensus state: current_term: 1 leader_uuid: "3dccd087fde7407ba6f27934f21a6b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } } }
I20260620 19:12:51.519775 24370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:51.519706 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dccd087fde7407ba6f27934f21a6b39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dccd087fde7407ba6f27934f21a6b39" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37413 } } }
I20260620 19:12:51.519865 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:51.519991 24378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:51.520635 24378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:51.521903 24378 catalog_manager.cc:1347] Generated new cluster ID: ffd4d40e8ea345779c893f86376ef868
I20260620 19:12:51.521955 24378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:51.533617 24378 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:51.533773 24378 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:51.536597 24378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: Generated new TSK 0
I20260620 19:12:51.536763 24378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:51.540624 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--builtin_ntp_servers=127.19.66.212:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:12:51.625360 24388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:51.625552 24388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:51.625581 24388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:51.626704 24388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:51.626741 24388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:51.626773 24388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:51.628069 24388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:51.628365 24388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:51.628690 24388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:51.633412 24393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.633564 24388 server_base.cc:1034] running on GCE node
W20260620 19:12:51.633415 24394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.633415 24396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.633828 24388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:51.634064 24388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:51.635202 24388 hybrid_clock.cc:648] HybridClock initialized: now 1781982771635189 us; error 29 us; skew 500 ppm
I20260620 19:12:51.636384 24388 webserver.cc:460] Webserver started at http://127.19.66.193:46877/ using document root <none> and password file <none>
I20260620 19:12:51.636612 24388 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:51.636662 24388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:51.636782 24388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:51.637837 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "20a3a073e7d5ac8820ae94bce452dbd9"
server_key_iv: "19e45516a2a68ea82ad214b10e2ff479"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.638154 24388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "20a3a073e7d5ac8820ae94bce452dbd9"
server_key_iv: "19e45516a2a68ea82ad214b10e2ff479"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.639356 24388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:51.640002 24402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.640141 24388 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:51.640219 24388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "20a3a073e7d5ac8820ae94bce452dbd9"
server_key_iv: "19e45516a2a68ea82ad214b10e2ff479"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.640280 24388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.643491 24388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.643726 24388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.643850 24388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.644024 24388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:51.644294 24388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:51.644325 24388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.644351 24388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:51.644373 24388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.649554 24388 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:43065
I20260620 19:12:51.649616 24515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:43065 every 8 connection(s)
I20260620 19:12:51.649899 24388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:51.650806 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24388
I20260620 19:12:51.650880 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:51.651050 19723 external_mini_cluster.cc:1428] Setting key 0a898a59cdff86a20a84be96ce78f1f3
I20260620 19:12:51.651947 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--builtin_ntp_servers=127.19.66.212:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 19:12:51.652793 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.650529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:44975 (local address 127.19.66.254:37413)
0620 19:12:51.650579 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.650581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.650805 (+ 224us) server_negotiation.cc:407] Connection header received
0620 19:12:51.651341 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.651342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.651351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.651407 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.652573 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.652575 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.652575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.652579 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.652597 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.652602 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.652666 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.652713 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.652727 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:12:51.652987 24518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.650504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37413 (local address 127.19.66.193:44975)
0620 19:12:51.650664 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.650696 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.651245 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.651465 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.651468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.651476 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.652432 (+ 956us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.652432 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.652462 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.652490 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.652664 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.652666 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.652668 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.652717 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.652763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260620 19:12:51.653378 24516 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37413
I20260620 19:12:51.653471 24516 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:51.653610 24516 heartbeater.cc:511] Master 127.19.66.254:37413 requested a full tablet report, sending...
I20260620 19:12:51.654094 24328 ts_manager.cc:194] Registered new tserver with Master: 97aa41cdb5ee4d338b8ba8f6f6d2943e (127.19.66.193:43065)
W20260620 19:12:51.729409 24519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:51.729596 24519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:51.729617 24519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:51.730758 24519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:51.730796 24519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:51.730821 24519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:51.732101 24519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:51.732318 24519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:51.732577 24519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:51.737135 24525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.737164 24524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.737164 24527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.737490 24519 server_base.cc:1034] running on GCE node
I20260620 19:12:51.737687 24519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:51.737905 24519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:51.739048 24519 hybrid_clock.cc:648] HybridClock initialized: now 1781982771739036 us; error 28 us; skew 500 ppm
I20260620 19:12:51.740211 24519 webserver.cc:460] Webserver started at http://127.19.66.194:36463/ using document root <none> and password file <none>
I20260620 19:12:51.740417 24519 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:51.740463 24519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:51.740586 24519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:51.741640 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "edd53e3d79d0498a952a86d8a44538e9"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "3c2a3a1d0c635860f8e9b780039d3558"
server_key_iv: "237058da0fee3f64debcfd3e63931f9c"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.741967 24519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "edd53e3d79d0498a952a86d8a44538e9"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "3c2a3a1d0c635860f8e9b780039d3558"
server_key_iv: "237058da0fee3f64debcfd3e63931f9c"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.743166 24519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:12:51.743793 24533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.743942 24519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:51.744020 24519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "edd53e3d79d0498a952a86d8a44538e9"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "3c2a3a1d0c635860f8e9b780039d3558"
server_key_iv: "237058da0fee3f64debcfd3e63931f9c"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.744082 24519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.747575 24519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.747798 24519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.747907 24519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.748085 24519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:51.748351 24519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:51.748382 24519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.748407 24519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:51.748431 24519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.753754 24519 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:42595
I20260620 19:12:51.753788 24646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:42595 every 8 connection(s)
I20260620 19:12:51.754062 24519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:51.756609 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.754676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:54761 (local address 127.19.66.254:37413)
0620 19:12:51.754727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.754729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.754924 (+ 195us) server_negotiation.cc:407] Connection header received
0620 19:12:51.755329 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.755330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.755335 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.755384 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.756436 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.756437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.756437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.756439 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.756451 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.756454 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.756485 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.756549 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.756559 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:12:51.756690 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24519
I20260620 19:12:51.756780 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:51.756816 24649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.754649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37413 (local address 127.19.66.194:54761)
0620 19:12:51.754815 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.754852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.755259 (+ 407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.755435 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.755439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.755446 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.756320 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.756321 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.756349 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.756376 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.756489 (+ 113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.756491 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.756494 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.756552 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.756611 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 19:12:51.756991 19723 external_mini_cluster.cc:1428] Setting key 160010372649724ad2c39daa29b71f72
I20260620 19:12:51.757230 24647 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37413
I20260620 19:12:51.757313 24647 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:51.757445 24647 heartbeater.cc:511] Master 127.19.66.254:37413 requested a full tablet report, sending...
I20260620 19:12:51.757788 24328 ts_manager.cc:194] Registered new tserver with Master: edd53e3d79d0498a952a86d8a44538e9 (127.19.66.194:42595)
I20260620 19:12:51.757807 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--builtin_ntp_servers=127.19.66.212:37815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 19:12:51.835731 24650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:51.835918 24650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:51.835938 24650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:51.837081 24650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:51.837117 24650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:51.837144 24650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:51.838433 24650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:37815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:51.838634 24650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:51.838927 24650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:51.843305 24655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.843321 24658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:51.843372 24656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:51.843492 24650 server_base.cc:1034] running on GCE node
I20260620 19:12:51.843677 24650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:51.843888 24650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:51.845018 24650 hybrid_clock.cc:648] HybridClock initialized: now 1781982771845004 us; error 27 us; skew 500 ppm
I20260620 19:12:51.846097 24650 webserver.cc:460] Webserver started at http://127.19.66.195:42841/ using document root <none> and password file <none>
I20260620 19:12:51.846303 24650 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:51.846354 24650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:51.846475 24650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:51.847483 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "11aca16fdc4044548fc660be78a4ce3c"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "0e0b04aa077114114c31d018752d85cb"
server_key_iv: "cb543b343c6550d5e6d31896e915e8b9"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.847833 24650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "11aca16fdc4044548fc660be78a4ce3c"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "0e0b04aa077114114c31d018752d85cb"
server_key_iv: "cb543b343c6550d5e6d31896e915e8b9"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.848960 24650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:51.849555 24664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.849699 24650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:51.849773 24650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "11aca16fdc4044548fc660be78a4ce3c"
format_stamp: "Formatted at 2026-06-20 19:12:51 on dist-test-slave-nq8r"
server_key: "0e0b04aa077114114c31d018752d85cb"
server_key_iv: "cb543b343c6550d5e6d31896e915e8b9"
server_key_version: "encryptionkey@0"
I20260620 19:12:51.849841 24650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:51.852947 24650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:51.853173 24650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:51.853281 24650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:51.853454 24650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:51.853713 24650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:51.853744 24650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.853767 24650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:51.853790 24650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:51.858994 24650 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33377
I20260620 19:12:51.859032 24777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33377 every 8 connection(s)
I20260620 19:12:51.859314 24650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:51.861918 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24650
I20260620 19:12:51.861953 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.859935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:35963 (local address 127.19.66.254:37413)
0620 19:12:51.859994 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.859996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.860210 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:12:51.860632 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.860633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.860638 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.860695 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.861731 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.861732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.861732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.861736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.861751 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.861754 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.861800 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.861886 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.861898 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:12:51.862015 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:51.862192 19723 external_mini_cluster.cc:1428] Setting key 24212e802d5b3e3b661bfa325f07afe1
I20260620 19:12:51.862226 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.859918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37413 (local address 127.19.66.195:35963)
0620 19:12:51.860093 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.860131 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.860560 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.860756 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.860760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.860767 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.861597 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.861597 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.861624 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.861652 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.861810 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.861814 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.861818 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.861897 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.861961 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260620 19:12:51.862633 24778 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37413
I20260620 19:12:51.862721 24778 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:51.862854 24778 heartbeater.cc:511] Master 127.19.66.254:37413 requested a full tablet report, sending...
I20260620 19:12:51.863188 24327 ts_manager.cc:194] Registered new tserver with Master: 11aca16fdc4044548fc660be78a4ce3c (127.19.66.195:33377)
I20260620 19:12:51.873920 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:12:51.876475 24368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.875373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.19.66.254:37413)
0620 19:12:51.875410 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.875413 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.875544 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:12:51.875618 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.875619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.875624 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.875664 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.876259 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.876261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.876261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.876265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.876281 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.876285 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.876332 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.876388 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.876402 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:12:51.878942 24327 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:12:51.881227 24327 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:12:51.885102 24787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.883472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33377 (local address 127.0.0.1:60862)
0620 19:12:51.883609 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.883663 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.883716 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.883837 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.883841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.883846 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.884656 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.884656 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.884672 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.884695 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.884942 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.884961 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.884964 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.885025 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.885038 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
I20260620 19:12:51.885296 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.883565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60862 (local address 127.19.66.195:33377)
0620 19:12:51.883719 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.883725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.883736 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:12:51.883761 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.883764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.883773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.883836 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.884786 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.884793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.884793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.884798 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.884838 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.884885 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.884936 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.885204 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.885221 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260620 19:12:51.885301 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.883486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42595 (local address 127.0.0.1:50790)
0620 19:12:51.884184 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.884193 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.884283 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.884463 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.884464 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.884467 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.884978 (+ 511us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.884978 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.884984 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.884995 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.885204 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.885204 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.885205 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.885242 (+ 37us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.885253 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":67,"threads_started":1}
I20260620 19:12:51.885483 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.883689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.19.66.194:42595)
0620 19:12:51.883843 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.883848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.884254 (+ 406us) server_negotiation.cc:407] Connection header received
0620 19:12:51.884369 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.884373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.884384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.884458 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.885067 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.885074 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.885074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.885078 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.885109 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.885132 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.885181 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.885391 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.885406 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 19:12:51.886149 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.884013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.19.66.193:43065)
0620 19:12:51.884434 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:51.884438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:51.884571 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:12:51.884596 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:51.884599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:51.884607 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:51.884667 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:51.885775 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:51.885782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:51.885782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:51.885787 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:51.885819 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:51.885842 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:51.885904 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:51.886047 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 19:12:51.886065 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":83,"threads_started":1}
I20260620 19:12:51.886070 24712 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f11d4a32d485bac083e271467468c (DEFAULT_TABLE table=test-table [id=39bf13af063b4e8a9db40e5ac6354381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.886463 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:51.883923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43065 (local address 127.0.0.1:60450)
0620 19:12:51.884480 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:51.884488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:51.884525 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:51.884968 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:51.884969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:51.884972 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:51.885669 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:51.885670 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:51.885677 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:51.885689 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:51.885907 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:51.885907 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:51.885908 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:12:51.886384 (+ 476us) client_negotiation.cc:240] Negotiation successful
0620 19:12:51.886398 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":80,"threads_started":1}
I20260620 19:12:51.886456 24712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f11d4a32d485bac083e271467468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.887037 24450 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f11d4a32d485bac083e271467468c (DEFAULT_TABLE table=test-table [id=39bf13af063b4e8a9db40e5ac6354381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.887336 24450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f11d4a32d485bac083e271467468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.889808 24793 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Bootstrap starting.
I20260620 19:12:51.889748 24792 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Bootstrap starting.
I20260620 19:12:51.889875 24581 tablet_service.cc:1467] Processing CreateTablet for tablet 1d7f11d4a32d485bac083e271467468c (DEFAULT_TABLE table=test-table [id=39bf13af063b4e8a9db40e5ac6354381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:12:51.890195 24581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d7f11d4a32d485bac083e271467468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:51.890444 24792 tablet_bootstrap.cc:654] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.890453 24793 tablet_bootstrap.cc:654] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.890692 24792 log.cc:824] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.890733 24793 log.cc:824] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.891330 24792 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: No bootstrap required, opened a new log
I20260620 19:12:51.891393 24792 ts_tablet_manager.cc:1397] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:12:51.893261 24796 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Bootstrap starting.
I20260620 19:12:51.893666 24792 raft_consensus.cc:348] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.893838 24792 raft_consensus.cc:374] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.893873 24792 raft_consensus.cc:729] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11aca16fdc4044548fc660be78a4ce3c, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.893926 24796 tablet_bootstrap.cc:654] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:51.893992 24792 consensus_queue.cc:260] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.894204 24796 log.cc:824] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:51.894250 24792 ts_tablet_manager.cc:1428] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:51.894274 24778 heartbeater.cc:503] Master 127.19.66.254:37413 was elected leader, sending a full tablet report...
I20260620 19:12:51.894681 24793 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: No bootstrap required, opened a new log
I20260620 19:12:51.894748 24793 ts_tablet_manager.cc:1397] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:12:51.895184 24796 tablet_bootstrap.cc:492] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: No bootstrap required, opened a new log
I20260620 19:12:51.895350 24796 ts_tablet_manager.cc:1397] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:12:51.897166 24793 raft_consensus.cc:348] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.897349 24793 raft_consensus.cc:374] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.897382 24793 raft_consensus.cc:729] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97aa41cdb5ee4d338b8ba8f6f6d2943e, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.897492 24793 consensus_queue.cc:260] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.897728 24793 ts_tablet_manager.cc:1428] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:51.897871 24516 heartbeater.cc:503] Master 127.19.66.254:37413 was elected leader, sending a full tablet report...
I20260620 19:12:51.898072 24796 raft_consensus.cc:348] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.898219 24796 raft_consensus.cc:374] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:51.898250 24796 raft_consensus.cc:729] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd53e3d79d0498a952a86d8a44538e9, State: Initialized, Role: FOLLOWER
I20260620 19:12:51.898365 24796 consensus_queue.cc:260] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:51.898599 24796 ts_tablet_manager.cc:1428] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:12:51.898720 24647 heartbeater.cc:503] Master 127.19.66.254:37413 was elected leader, sending a full tablet report...
W20260620 19:12:51.900313 24517 tablet.cc:2369] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:52.004557 24648 tablet.cc:2369] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:12:52.109798 24779 tablet.cc:2369] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:12:52.251470 24799 raft_consensus.cc:482] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:52.251608 24799 raft_consensus.cc:504] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:52.252002 24799 leader_election.cc:288] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11aca16fdc4044548fc660be78a4ce3c (127.19.66.195:33377), edd53e3d79d0498a952a86d8a44538e9 (127.19.66.194:42595)
I20260620 19:12:52.256642 24797 raft_consensus.cc:482] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:12:52.256748 24797 raft_consensus.cc:504] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:52.257072 24797 leader_election.cc:288] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd53e3d79d0498a952a86d8a44538e9 (127.19.66.194:42595), 97aa41cdb5ee4d338b8ba8f6f6d2943e (127.19.66.193:43065)
I20260620 19:12:52.258322 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.257212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43065 (local address 127.19.66.195:43951)
0620 19:12:52.257268 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.257277 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.257312 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.257515 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.257516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.257518 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:52.258038 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:52.258038 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:52.258045 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:52.258059 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:52.258204 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:52.258205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:52.258205 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:12:52.258244 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:12:52.258258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:12:52.258440 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.252258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:47271 (local address 127.19.66.194:42595)
0620 19:12:52.252311 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.252313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.252426 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:12:52.252474 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.252475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.252480 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.252532 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.258246 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.258247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.258248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.258252 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.258268 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.258271 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.258315 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.258361 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.258374 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:12:52.258453 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.252280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:34403 (local address 127.19.66.195:33377)
0620 19:12:52.252323 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.252325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.252391 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:12:52.252439 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.252440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.252446 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.252498 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.258111 (+ 5613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.258112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.258112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.258116 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.258130 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.258135 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.258165 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.258390 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.258402 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:12:52.258459 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.252188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42595 (local address 127.19.66.193:47271)
0620 19:12:52.252338 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.252348 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.252423 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.252548 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.252549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.252564 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:52.258166 (+ 5602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:52.258166 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:52.258173 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:52.258186 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:52.258322 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:52.258323 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:52.258323 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:12:52.258364 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:12:52.258375 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":53,"threads_started":1}
I20260620 19:12:52.258590 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.252182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33377 (local address 127.19.66.193:34403)
0620 19:12:52.252297 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.252309 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.252387 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.252510 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.252510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.252513 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:52.257994 (+ 5481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:52.257994 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:52.258003 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:52.258018 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:52.258204 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:52.258205 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:52.258205 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 19:12:52.258507 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 19:12:52.258519 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260620 19:12:52.258747 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.257215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42595 (local address 127.19.66.195:33861)
0620 19:12:52.257365 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.257375 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.257430 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.257546 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.257547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.257549 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:52.258282 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:12:52.258282 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:52.258287 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:12:52.258296 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:52.258628 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:52.258628 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:52.258629 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:12:52.258668 (+ 39us) client_negotiation.cc:240] Negotiation successful
0620 19:12:52.258679 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":21,"thread_start_us":72,"threads_started":1}
I20260620 19:12:52.258831 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aca16fdc4044548fc660be78a4ce3c" is_pre_election: true
I20260620 19:12:52.258926 24732 raft_consensus.cc:2454] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97aa41cdb5ee4d338b8ba8f6f6d2943e in term 0.
I20260620 19:12:52.258860 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd53e3d79d0498a952a86d8a44538e9" is_pre_election: true
I20260620 19:12:52.258993 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.257251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:33861 (local address 127.19.66.194:42595)
0620 19:12:52.257366 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.257367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.257424 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:12:52.257470 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.257471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.257476 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.257537 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.258534 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.258535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.258536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.258539 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.258552 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.258556 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.258603 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.258818 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.258832 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260620 19:12:52.259142 24403 leader_election.cc:302] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aca16fdc4044548fc660be78a4ce3c, 97aa41cdb5ee4d338b8ba8f6f6d2943e; no voters:
I20260620 19:12:52.259212 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.257374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:43951 (local address 127.19.66.193:43065)
0620 19:12:52.257418 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.257419 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.257426 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:52.257447 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.257448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.257453 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.257510 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.258117 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.258118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.258118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.258122 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.258131 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.258135 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.258181 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.259147 (+ 966us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.259160 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:12:52.259019 24601 raft_consensus.cc:2454] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97aa41cdb5ee4d338b8ba8f6f6d2943e in term 0.
I20260620 19:12:52.259259 24799 raft_consensus.cc:2781] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:52.259256 24600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "11aca16fdc4044548fc660be78a4ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd53e3d79d0498a952a86d8a44538e9" is_pre_election: true
I20260620 19:12:52.259317 24799 raft_consensus.cc:482] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:52.259339 24799 raft_consensus.cc:3037] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.259331 24600 raft_consensus.cc:2454] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11aca16fdc4044548fc660be78a4ce3c in term 0.
I20260620 19:12:52.259370 24470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "11aca16fdc4044548fc660be78a4ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" is_pre_election: true
I20260620 19:12:52.259503 24667 leader_election.cc:302] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11aca16fdc4044548fc660be78a4ce3c, edd53e3d79d0498a952a86d8a44538e9; no voters:
I20260620 19:12:52.259617 24797 raft_consensus.cc:2781] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:12:52.259650 24797 raft_consensus.cc:482] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:12:52.259667 24797 raft_consensus.cc:3037] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.260226 24799 raft_consensus.cc:504] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:52.260363 24470 raft_consensus.cc:2379] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 11aca16fdc4044548fc660be78a4ce3c in current term 1: Already voted for candidate 97aa41cdb5ee4d338b8ba8f6f6d2943e in this term.
I20260620 19:12:52.260435 24797 raft_consensus.cc:504] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:52.260557 24797 leader_election.cc:288] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [CANDIDATE]: Term 1 election: Requested vote from peers edd53e3d79d0498a952a86d8a44538e9 (127.19.66.194:42595), 97aa41cdb5ee4d338b8ba8f6f6d2943e (127.19.66.193:43065)
I20260620 19:12:52.260694 24600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "11aca16fdc4044548fc660be78a4ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd53e3d79d0498a952a86d8a44538e9"
I20260620 19:12:52.260701 24469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "11aca16fdc4044548fc660be78a4ce3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e"
I20260620 19:12:52.260758 24600 raft_consensus.cc:3037] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.260790 24469 raft_consensus.cc:2379] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 11aca16fdc4044548fc660be78a4ce3c in current term 1: Already voted for candidate 97aa41cdb5ee4d338b8ba8f6f6d2943e in this term.
I20260620 19:12:52.260969 24799 leader_election.cc:288] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [CANDIDATE]: Term 1 election: Requested vote from peers 11aca16fdc4044548fc660be78a4ce3c (127.19.66.195:33377), edd53e3d79d0498a952a86d8a44538e9 (127.19.66.194:42595)
I20260620 19:12:52.261036 24732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11aca16fdc4044548fc660be78a4ce3c"
I20260620 19:12:52.261119 24732 raft_consensus.cc:2379] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 97aa41cdb5ee4d338b8ba8f6f6d2943e in current term 1: Already voted for candidate 11aca16fdc4044548fc660be78a4ce3c in this term.
I20260620 19:12:52.261296 24601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d7f11d4a32d485bac083e271467468c" candidate_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd53e3d79d0498a952a86d8a44538e9"
I20260620 19:12:52.261526 24405 leader_election.cc:302] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97aa41cdb5ee4d338b8ba8f6f6d2943e; no voters: 11aca16fdc4044548fc660be78a4ce3c, edd53e3d79d0498a952a86d8a44538e9
I20260620 19:12:52.261601 24600 raft_consensus.cc:2454] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11aca16fdc4044548fc660be78a4ce3c in term 1.
I20260620 19:12:52.261611 24799 raft_consensus.cc:2726] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 19:12:52.261726 24667 leader_election.cc:302] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11aca16fdc4044548fc660be78a4ce3c, edd53e3d79d0498a952a86d8a44538e9; no voters: 97aa41cdb5ee4d338b8ba8f6f6d2943e
I20260620 19:12:52.261821 24797 raft_consensus.cc:2781] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:52.261961 24797 raft_consensus.cc:686] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 LEADER]: Becoming Leader. State: Replica: 11aca16fdc4044548fc660be78a4ce3c, State: Running, Role: LEADER
I20260620 19:12:52.262040 24797 consensus_queue.cc:237] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } }
I20260620 19:12:52.262800 24327 catalog_manager.cc:5495] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 11aca16fdc4044548fc660be78a4ce3c (127.19.66.195). New cstate: current_term: 1 leader_uuid: "11aca16fdc4044548fc660be78a4ce3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11aca16fdc4044548fc660be78a4ce3c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 } health_report { overall_health: UNKNOWN } } }
I20260620 19:12:52.325441 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.324490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60878 (local address 127.19.66.195:33377)
0620 19:12:52.324535 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.324537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.324544 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:52.324585 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.324585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.324604 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.324652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.325218 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.325220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.325220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.325224 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.325239 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.325244 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.325295 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.325352 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.325365 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:12:52.327153 24469 raft_consensus.cc:1261] T 1d7f11d4a32d485bac083e271467468c P 97aa41cdb5ee4d338b8ba8f6f6d2943e [term 1 FOLLOWER]: Refusing update from remote peer 11aca16fdc4044548fc660be78a4ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:52.327157 24600 raft_consensus.cc:1261] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 1 FOLLOWER]: Refusing update from remote peer 11aca16fdc4044548fc660be78a4ce3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:12:52.327387 24797 consensus_queue.cc:1035] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd53e3d79d0498a952a86d8a44538e9" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:52.327524 24805 consensus_queue.cc:1035] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "97aa41cdb5ee4d338b8ba8f6f6d2943e" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:12:52.329373 24808 mvcc.cc:205] Tried to move back new op lower bound from 7299001435449380864 to 7299001435185741824. Current Snapshot: MvccSnapshot[applied={T|T < 7299001435449380864}]
I20260620 19:12:52.329752 24809 mvcc.cc:205] Tried to move back new op lower bound from 7299001435449380864 to 7299001435185741824. Current Snapshot: MvccSnapshot[applied={T|T < 7299001435449380864}]
I20260620 19:12:52.332265 24327 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:52.332358 24327 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:12:52.333019 24327 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3dccd087fde7407ba6f27934f21a6b39: Sending DeleteTablet for 3 replicas of tablet 1d7f11d4a32d485bac083e271467468c
I20260620 19:12:52.333326 24581 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7f11d4a32d485bac083e271467468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:52 UTC) from {username='slave'} at 127.0.0.1:50790
I20260620 19:12:52.333350 24712 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7f11d4a32d485bac083e271467468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:52 UTC) from {username='slave'} at 127.0.0.1:60862
I20260620 19:12:52.333357 24450 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d7f11d4a32d485bac083e271467468c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:12:52 UTC) from {username='slave'} at 127.0.0.1:60450
I20260620 19:12:52.333578 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24388
I20260620 19:12:52.336966 24818 tablet_replica.cc:330] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: stopping tablet replica
I20260620 19:12:52.337077 24818 raft_consensus.cc:2229] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:12:52.337189 24818 raft_consensus.cc:2258] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:52.337347 24817 tablet_replica.cc:330] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: stopping tablet replica
I20260620 19:12:52.337440 24817 raft_consensus.cc:2229] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:12:52.337551 24817 raft_consensus.cc:2258] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:12:52.338030 24818 ts_tablet_manager.cc:1905] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:52.338028 24817 ts_tablet_manager.cc:1905] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:12:52.339630 24817 ts_tablet_manager.cc:1918] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:52.339630 24818 ts_tablet_manager.cc:1918] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:12:52.339700 24818 log.cc:1196] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/1d7f11d4a32d485bac083e271467468c
I20260620 19:12:52.339807 24817 log.cc:1196] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/1d7f11d4a32d485bac083e271467468c
I20260620 19:12:52.339965 24818 ts_tablet_manager.cc:1939] T 1d7f11d4a32d485bac083e271467468c P edd53e3d79d0498a952a86d8a44538e9: Deleting consensus metadata
I20260620 19:12:52.340183 24817 ts_tablet_manager.cc:1939] T 1d7f11d4a32d485bac083e271467468c P 11aca16fdc4044548fc660be78a4ce3c: Deleting consensus metadata
W20260620 19:12:52.340178 24315 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:43065 (error 108)
I20260620 19:12:52.340431 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24519
I20260620 19:12:52.341600 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.341458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43065 (local address 127.0.0.1:60456)
0620 19:12:52.341520 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.341566 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:43065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:12:52.341761 24315 catalog_manager.cc:4584] TS 97aa41cdb5ee4d338b8ba8f6f6d2943e (127.19.66.193:43065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d7f11d4a32d485bac083e271467468c: Network error: Client connection negotiation failed: client connection to 127.19.66.193:43065: connect: Connection refused (error 111)
I20260620 19:12:52.345352 24312 catalog_manager.cc:4841] TS 11aca16fdc4044548fc660be78a4ce3c (127.19.66.195:33377): tablet 1d7f11d4a32d485bac083e271467468c (table test-table [id=39bf13af063b4e8a9db40e5ac6354381]) successfully deleted
I20260620 19:12:52.345422 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.345323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:42595 (local address 127.0.0.1:50804)
0620 19:12:52.345368 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.345392 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.194:42595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:12:52.345625 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24650
I20260620 19:12:52.350220 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24297
2026-06-20T19:12:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (962 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 19:12:52.368294 19723 mini_oidc.cc:152] Starting JWKS server
I20260620 19:12:52.369458 19723 webserver.cc:460] Webserver started at https://127.0.0.1:43559/ using document root <none> and password file <none>
I20260620 19:12:52.369545 19723 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:12:52.369714 19723 webserver.cc:460] Webserver started at http://127.0.0.1:41559/ using document root <none> and password file <none>
2026-06-20T19:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:52Z Disabled control of system clock
I20260620 19:12:52.376725 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43341
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/testchainca.cert with env {}
W20260620 19:12:52.454515 24826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:52.454704 24826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:52.454723 24826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:52.455811 24826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:52.455844 24826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:52.455855 24826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:52.455865 24826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:52.455878 24826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:12:52.455894 24826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:12:52.455904 24826 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43559/jwks.json
I20260620 19:12:52.457458 24826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:52.457640 24826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:52.457810 24826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:52.462108 24832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.462190 24831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.462138 24834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:52.462385 24826 server_base.cc:1034] running on GCE node
I20260620 19:12:52.462610 24826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:52.462874 24826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:52.464020 24826 hybrid_clock.cc:648] HybridClock initialized: now 1781982772464005 us; error 30 us; skew 500 ppm
I20260620 19:12:52.465206 24826 webserver.cc:460] Webserver started at http://127.19.66.254:43083/ using document root <none> and password file <none>
I20260620 19:12:52.465401 24826 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:52.465440 24826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:52.465525 24826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:52.466437 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "fe5077461ff740f685ba9f30c958fda1"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "817a849972f218a8817d97e396e0c808"
server_key_iv: "40c3d99a15552d46d7d64b15dff3108b"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.466710 24826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "fe5077461ff740f685ba9f30c958fda1"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "817a849972f218a8817d97e396e0c808"
server_key_iv: "40c3d99a15552d46d7d64b15dff3108b"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.467795 24826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:52.468391 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:52.468566 24826 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:52.468645 24826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "fe5077461ff740f685ba9f30c958fda1"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "817a849972f218a8817d97e396e0c808"
server_key_iv: "40c3d99a15552d46d7d64b15dff3108b"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.468698 24826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:52.477429 24826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:52.477653 24826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:52.477743 24826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:52.481184 24826 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:43341
I20260620 19:12:52.481245 24892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:43341 every 8 connection(s)
I20260620 19:12:52.481513 24826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:52.482407 24893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:52.484613 24893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1: Bootstrap starting.
I20260620 19:12:52.485213 24893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:52.485487 24893 log.cc:824] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:52.486073 24893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1: No bootstrap required, opened a new log
I20260620 19:12:52.487578 24893 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } }
I20260620 19:12:52.487715 24893 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:52.487754 24893 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5077461ff740f685ba9f30c958fda1, State: Initialized, Role: FOLLOWER
I20260620 19:12:52.487900 24893 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } }
I20260620 19:12:52.487977 24893 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:52.488008 24893 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:52.488054 24893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.488543 24893 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } }
I20260620 19:12:52.488634 24893 leader_election.cc:302] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe5077461ff740f685ba9f30c958fda1; no voters:
I20260620 19:12:52.488806 24893 leader_election.cc:288] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:52.488880 24896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:52.489035 24896 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [term 1 LEADER]: Becoming Leader. State: Replica: fe5077461ff740f685ba9f30c958fda1, State: Running, Role: LEADER
I20260620 19:12:52.489064 24893 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:52.489159 24896 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } }
I20260620 19:12:52.489538 24897 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe5077461ff740f685ba9f30c958fda1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } } }
I20260620 19:12:52.489619 24897 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.489619 24898 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe5077461ff740f685ba9f30c958fda1. Latest consensus state: current_term: 1 leader_uuid: "fe5077461ff740f685ba9f30c958fda1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe5077461ff740f685ba9f30c958fda1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43341 } } }
I20260620 19:12:52.489719 24898 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.489959 24905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:52.490350 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24826
I20260620 19:12:52.490440 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:52.490465 24905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:52.490619 19723 external_mini_cluster.cc:1428] Setting key ab50aeb358d83282ab57bdc9bccae222
I20260620 19:12:52.492040 24905 catalog_manager.cc:1347] Generated new cluster ID: 62263f8f7b284df9b4dd6a867f03a6de
I20260620 19:12:52.492092 24905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:52.496678 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.491028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58240 (local address 127.19.66.254:43341)
0620 19:12:52.491522 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.491528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.491554 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:12:52.491671 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.491679 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.491745 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.491864 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.492064 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.493413 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.494546 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.495164 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.495261 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.495990 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.495994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.495994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.496205 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.496244 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.496258 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.496316 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.496395 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.496468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":75,"threads_started":1}
I20260620 19:12:52.497578 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:52.502466 24905 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:52.503123 24905 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:52.507604 24905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe5077461ff740f685ba9f30c958fda1: Generated new TSK 0
I20260620 19:12:52.507767 24905 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:12:52.542258 24923 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 19:12:52.542356 24923 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:12:52.542500 24915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.519862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58244 (local address 127.19.66.254:43341)
0620 19:12:52.519923 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.519925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.520058 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:12:52.520158 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.520158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.520168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.520222 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:12:52.520363 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.520936 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.522027 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.522204 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.522247 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.522440 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:12:52.542258 (+ 19818us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:12:52.542422 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 19:12:52.542552 24915 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58244: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 19:12:52.542862 24922 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:52.519798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43341 (local address 127.0.0.1:58244)
0620 19:12:52.519989 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.519999 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.520072 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.520228 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.520229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.520231 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:12:52.520307 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.520308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.521002 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.521003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.521955 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.521958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.522267 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.522268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.522327 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.522349 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:12:52.542400 (+ 20051us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 19:12:52.542515 (+ 115us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.66.254:43341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.19.66.254:43341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 19:12:52.543224 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24826
2026-06-20T19:12:52Z chronyd exiting
I20260620 19:12:52.578431 19723 test_util.cc:175] -----------------------------------------------
I20260620 19:12:52.578506 19723 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781982751812665-19723-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 19:12:52.579188 19723 mini_oidc.cc:152] Starting JWKS server
I20260620 19:12:52.579809 19723 webserver.cc:460] Webserver started at https://127.0.0.1:44277/ using document root <none> and password file <none>
I20260620 19:12:52.579856 19723 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:12:52.579963 19723 webserver.cc:460] Webserver started at http://127.0.0.1:44575/ using document root <none> and password file <none>
2026-06-20T19:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:52Z Disabled control of system clock
I20260620 19:12:52.586869 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:34297
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:42465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:34297
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44277/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/testchainca.cert with env {}
W20260620 19:12:52.663774 24929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:52.663966 24929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:52.663987 24929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:52.665079 24929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:52.665112 24929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:52.665122 24929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:52.665132 24929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:52.665148 24929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:12:52.665164 24929 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:12:52.665174 24929 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44277/jwks.json
I20260620 19:12:52.666582 24929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:34297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44277/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:52.666755 24929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:52.666910 24929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:52.671123 24937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.671200 24934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.671149 24935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:52.671268 24929 server_base.cc:1034] running on GCE node
I20260620 19:12:52.671531 24929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:52.671803 24929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:52.672899 24929 hybrid_clock.cc:648] HybridClock initialized: now 1781982772672884 us; error 29 us; skew 500 ppm
I20260620 19:12:52.674010 24929 webserver.cc:460] Webserver started at http://127.19.66.254:41669/ using document root <none> and password file <none>
I20260620 19:12:52.674254 24929 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:52.674301 24929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:52.674427 24929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:52.675424 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "007d89bebc0b42659c104ee2f1c2dd85"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "c4af163dda542b31cdb605825ed5392e"
server_key_iv: "0701acaec151b405f62608f5b8fc120f"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.675750 24929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "007d89bebc0b42659c104ee2f1c2dd85"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "c4af163dda542b31cdb605825ed5392e"
server_key_iv: "0701acaec151b405f62608f5b8fc120f"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.676975 24929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:52.677646 24943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:52.677809 24929 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:52.677891 24929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "007d89bebc0b42659c104ee2f1c2dd85"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "c4af163dda542b31cdb605825ed5392e"
server_key_iv: "0701acaec151b405f62608f5b8fc120f"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.677959 24929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:52.689790 24929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:52.690085 24929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:52.690207 24929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:52.693784 24929 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:34297
I20260620 19:12:52.693820 24995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:34297 every 8 connection(s)
I20260620 19:12:52.694141 24929 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:52.694758 24996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:52.697506 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85: Bootstrap starting.
I20260620 19:12:52.698047 24996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:52.698335 24996 log.cc:824] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:52.699045 24996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85: No bootstrap required, opened a new log
I20260620 19:12:52.700556 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 24929
I20260620 19:12:52.700641 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:52.700621 24996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } }
I20260620 19:12:52.700775 24996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:52.700802 19723 external_mini_cluster.cc:1428] Setting key ee853c17f07e011be79c2fa874ff1304
I20260620 19:12:52.700802 24996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 007d89bebc0b42659c104ee2f1c2dd85, State: Initialized, Role: FOLLOWER
I20260620 19:12:52.700965 24996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } }
I20260620 19:12:52.701046 24996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:52.701082 24996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:52.701129 24996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.701766 24996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } }
I20260620 19:12:52.701882 24996 leader_election.cc:302] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 007d89bebc0b42659c104ee2f1c2dd85; no voters:
I20260620 19:12:52.702034 24996 leader_election.cc:288] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:52.702059 25001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:52.702185 25001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [term 1 LEADER]: Becoming Leader. State: Replica: 007d89bebc0b42659c104ee2f1c2dd85, State: Running, Role: LEADER
I20260620 19:12:52.702271 24996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:52.702281 25001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } }
I20260620 19:12:52.702670 25001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 007d89bebc0b42659c104ee2f1c2dd85. Latest consensus state: current_term: 1 leader_uuid: "007d89bebc0b42659c104ee2f1c2dd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } } }
I20260620 19:12:52.702744 25001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.702900 25003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "007d89bebc0b42659c104ee2f1c2dd85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "007d89bebc0b42659c104ee2f1c2dd85" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34297 } } }
I20260620 19:12:52.702961 25003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.703068 25008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:52.703554 25008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:52.705032 25008 catalog_manager.cc:1347] Generated new cluster ID: 6feefdadc289443e9193a4bf951bc04a
I20260620 19:12:52.705092 25008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:52.706622 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.701232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.19.66.254:34297)
0620 19:12:52.701451 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.701461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.701494 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:12:52.701620 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.701628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.701707 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.701861 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.702024 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.703813 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.704648 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.705247 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.705356 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.706009 (+ 653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.706013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.706014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.706175 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.706207 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.706218 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.706272 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.706385 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.706464 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260620 19:12:52.707408 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:52.712697 25008 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:52.713372 25008 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:52.721081 25008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 007d89bebc0b42659c104ee2f1c2dd85: Generated new TSK 0
I20260620 19:12:52.721262 25008 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 19:12:52.743221 25026 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 19:12:52.743304 25026 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 19:12:52.743595 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.729416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.19.66.254:34297)
0620 19:12:52.729468 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.729470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.729566 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:12:52.729642 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.729643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.729653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.729708 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:12:52.729824 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.730451 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.731577 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.731793 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.731852 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.732009 (+ 157us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:12:52.743410 (+ 11401us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:12:52.743545 (+ 135us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 19:12:52.743639 25000 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42876: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 19:12:52.743739 25025 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:52.729357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:34297 (local address 127.0.0.1:42876)
0620 19:12:52.729500 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.729510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.729576 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.729709 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.729710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.729712 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:12:52.729770 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.729772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.730530 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.730531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.731477 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.731480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.731867 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.731868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.731925 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.731930 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:12:52.743516 (+ 11586us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 19:12:52.743639 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.66.254:34297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 19:12:52.744009 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 24929
2026-06-20T19:12:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (201 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 19:12:52.780584 19723 mini_oidc.cc:152] Starting JWKS server
I20260620 19:12:52.781292 19723 webserver.cc:460] Webserver started at https://127.0.0.1:45419/ using document root <none> and password file <none>
I20260620 19:12:52.781339 19723 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 19:12:52.781471 19723 webserver.cc:460] Webserver started at http://127.0.0.1:46593/ using document root <none> and password file <none>
2026-06-20T19:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:52Z Disabled control of system clock
I20260620 19:12:52.788483 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:39717
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:46759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:39717
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45419/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:12:52.865689 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:52.865888 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:52.865907 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:52.867045 25032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:52.867082 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:52.867093 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:52.867105 25032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:52.867125 25032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:12:52.867148 25032 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:12:52.867168 25032 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45419/jwks.json
I20260620 19:12:52.868494 25032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:46759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:39717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45419/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:52.868696 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:52.868973 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:52.873211 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.873317 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:52.873410 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:52.873420 25032 server_base.cc:1034] running on GCE node
I20260620 19:12:52.873711 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:52.873953 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:52.875099 25032 hybrid_clock.cc:648] HybridClock initialized: now 1781982772875081 us; error 32 us; skew 500 ppm
I20260620 19:12:52.876302 25032 webserver.cc:460] Webserver started at http://127.19.66.254:43595/ using document root <none> and password file <none>
I20260620 19:12:52.876514 25032 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:52.876565 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:52.876689 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:52.877776 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "beaa3c8a55254b559a0562f8caaab13f"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "8582cb5189ea310aede06c79f9de1c8a"
server_key_iv: "4627e7e8c82f434cab2e5d48743e2cae"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.878101 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "beaa3c8a55254b559a0562f8caaab13f"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "8582cb5189ea310aede06c79f9de1c8a"
server_key_iv: "4627e7e8c82f434cab2e5d48743e2cae"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.879290 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:52.879912 25046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:52.880055 25032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:52.880131 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "beaa3c8a55254b559a0562f8caaab13f"
format_stamp: "Formatted at 2026-06-20 19:12:52 on dist-test-slave-nq8r"
server_key: "8582cb5189ea310aede06c79f9de1c8a"
server_key_iv: "4627e7e8c82f434cab2e5d48743e2cae"
server_key_version: "encryptionkey@0"
I20260620 19:12:52.880200 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:52.895815 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:52.896116 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:52.896265 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:52.899895 25032 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:39717
I20260620 19:12:52.899927 25098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:39717 every 8 connection(s)
I20260620 19:12:52.900233 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:52.900868 25099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:52.902226 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25032
I20260620 19:12:52.902318 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:52.902474 19723 external_mini_cluster.cc:1428] Setting key afa8e17ba3c01b20c7ca4653d3f436a0
I20260620 19:12:52.904012 25099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f: Bootstrap starting.
I20260620 19:12:52.904687 25099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:52.905021 25099 log.cc:824] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:52.905702 25099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f: No bootstrap required, opened a new log
I20260620 19:12:52.907755 25099 raft_consensus.cc:348] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } }
I20260620 19:12:52.907918 25099 raft_consensus.cc:374] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:52.907953 25099 raft_consensus.cc:729] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beaa3c8a55254b559a0562f8caaab13f, State: Initialized, Role: FOLLOWER
I20260620 19:12:52.908085 25099 consensus_queue.cc:260] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } }
I20260620 19:12:52.908146 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.902951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57664 (local address 127.19.66.254:39717)
0620 19:12:52.903159 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.903168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.903202 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:12:52.903747 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.903756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.903836 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.903989 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:52.904176 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.905428 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.906185 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.906787 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.906930 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.907498 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:52.907503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:52.907503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:52.907696 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:52.907737 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:52.907752 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:52.907818 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:52.907872 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:12:52.907950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260620 19:12:52.908164 25099 raft_consensus.cc:388] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:52.908192 25099 raft_consensus.cc:482] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:52.908229 25099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:52.908803 25099 raft_consensus.cc:504] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } }
I20260620 19:12:52.908927 25099 leader_election.cc:302] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beaa3c8a55254b559a0562f8caaab13f; no voters:
I20260620 19:12:52.908999 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:52.909076 25099 leader_election.cc:288] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:52.909114 25104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:52.909236 25104 raft_consensus.cc:686] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [term 1 LEADER]: Becoming Leader. State: Replica: beaa3c8a55254b559a0562f8caaab13f, State: Running, Role: LEADER
I20260620 19:12:52.909322 25104 consensus_queue.cc:237] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } }
I20260620 19:12:52.909322 25099 sys_catalog.cc:564] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:52.910267 25105 sys_catalog.cc:455] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [sys.catalog]: SysCatalogTable state changed. Reason: New leader beaa3c8a55254b559a0562f8caaab13f. Latest consensus state: current_term: 1 leader_uuid: "beaa3c8a55254b559a0562f8caaab13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } } }
I20260620 19:12:52.910336 25105 sys_catalog.cc:458] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.910456 25106 sys_catalog.cc:455] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beaa3c8a55254b559a0562f8caaab13f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beaa3c8a55254b559a0562f8caaab13f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 39717 } } }
I20260620 19:12:52.910494 25116 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:52.910528 25106 sys_catalog.cc:458] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:52.910882 25116 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:52.912220 25116 catalog_manager.cc:1347] Generated new cluster ID: efe5e23a558c4dd0a03c7c140623d6e9
I20260620 19:12:52.912271 25116 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:52.924104 25116 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:52.924813 25116 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:52.941555 25116 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beaa3c8a55254b559a0562f8caaab13f: Generated new TSK 0
I20260620 19:12:52.941777 25116 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:52.950337 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:52.946894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.19.66.254:39717)
0620 19:12:52.946951 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:52.946953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:52.947053 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:12:52.947130 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:52.947130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:52.947141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:52.947209 (+ 68us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:12:52.947352 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.947851 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.948963 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.949174 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.949228 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.949394 (+ 166us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:12:52.950142 (+ 748us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:12:52.950268 (+ 126us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 19:12:52.950400 25102 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57680: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 19:12:52.950441 25128 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:52.946832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:39717 (local address 127.0.0.1:57680)
0620 19:12:52.946983 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:52.946994 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:52.947065 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:52.947223 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:52.947224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:52.947225 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:12:52.947300 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.947301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.947919 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.947920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.948891 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:52.948895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:52.949250 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:52.949250 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:52.949308 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:52.949314 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:12:52.950235 (+ 921us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 19:12:52.950329 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.66.254:39717: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45419/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260620 19:12:52.950872 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25032
E20260620 19:12:52.961102 25129 webserver.cc:533] Webserver: sslize failed:
E20260620 19:12:52.961171 25129 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T19:12:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (206 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T19:12:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:52Z Disabled control of system clock
I20260620 19:12:52.994179 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:42041
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:38965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:42041
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/testchainca.cert with env {}
W20260620 19:12:53.071748 25133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:53.071964 25133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:53.071985 25133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:53.073135 25133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:53.073169 25133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:53.073180 25133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:53.073191 25133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:53.073201 25133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 19:12:53.073212 25133 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 19:12:53.073222 25133 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 19:12:53.074679 25133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:42041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:42041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:53.074887 25133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:53.075054 25133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:53.079563 25141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:53.079586 25138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:53.079725 25133 server_base.cc:1034] running on GCE node
W20260620 19:12:53.079571 25139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:53.080021 25133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:53.080309 25133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:53.081475 25133 hybrid_clock.cc:648] HybridClock initialized: now 1781982773081464 us; error 32 us; skew 500 ppm
I20260620 19:12:53.082671 25133 webserver.cc:460] Webserver started at http://127.19.66.254:42647/ using document root <none> and password file <none>
I20260620 19:12:53.082890 25133 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:53.082940 25133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:53.083066 25133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:53.084107 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "6f919694baf642efb52c582cffc9ccc2"
format_stamp: "Formatted at 2026-06-20 19:12:53 on dist-test-slave-nq8r"
server_key: "6be631c8bbc96734ea9cf3c3906ec02e"
server_key_iv: "1ef715a7e2d39c799fd38cf5b79c7999"
server_key_version: "encryptionkey@0"
I20260620 19:12:53.084474 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "6f919694baf642efb52c582cffc9ccc2"
format_stamp: "Formatted at 2026-06-20 19:12:53 on dist-test-slave-nq8r"
server_key: "6be631c8bbc96734ea9cf3c3906ec02e"
server_key_iv: "1ef715a7e2d39c799fd38cf5b79c7999"
server_key_version: "encryptionkey@0"
I20260620 19:12:53.085719 25133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:12:53.086350 25147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:53.086491 25133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:12:53.086572 25133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "6f919694baf642efb52c582cffc9ccc2"
format_stamp: "Formatted at 2026-06-20 19:12:53 on dist-test-slave-nq8r"
server_key: "6be631c8bbc96734ea9cf3c3906ec02e"
server_key_iv: "1ef715a7e2d39c799fd38cf5b79c7999"
server_key_version: "encryptionkey@0"
I20260620 19:12:53.086628 25133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:53.102602 25133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:53.102905 25133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:53.103027 25133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:53.106740 25133 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:42041
I20260620 19:12:53.106748 25199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:42041 every 8 connection(s)
I20260620 19:12:53.107115 25133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:53.107882 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25133
I20260620 19:12:53.107975 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:53.108134 19723 external_mini_cluster.cc:1428] Setting key 41cc1be291e34d1ec0b6d9e9ba44ea04
I20260620 19:12:53.108129 25200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:53.111577 25200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: Bootstrap starting.
I20260620 19:12:53.112202 25200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:53.112496 25200 log.cc:824] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:53.113284 25200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: No bootstrap required, opened a new log
I20260620 19:12:53.114413 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:53.108641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.19.66.254:42041)
0620 19:12:53.108891 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:53.108901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:53.108937 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:12:53.109623 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:53.109634 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:53.109725 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:53.109897 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:53.110114 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.111436 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.112289 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.112895 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.113012 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:53.113672 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:53.113677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:53.113677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:12:53.113887 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:53.113928 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:12:53.113944 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:53.114010 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:53.114103 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:12:53.114189 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260620 19:12:53.115351 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:12:53.115514 25200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } }
I20260620 19:12:53.115650 25200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:53.115682 25200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f919694baf642efb52c582cffc9ccc2, State: Initialized, Role: FOLLOWER
I20260620 19:12:53.115800 25200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } }
I20260620 19:12:53.115872 25200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:53.115895 25200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:53.115940 25200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:53.116562 25200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } }
I20260620 19:12:53.116652 25200 leader_election.cc:302] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f919694baf642efb52c582cffc9ccc2; no voters:
I20260620 19:12:53.116816 25200 leader_election.cc:288] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:53.116887 25205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:53.117027 25205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f919694baf642efb52c582cffc9ccc2, State: Running, Role: LEADER
I20260620 19:12:53.117100 25200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:53.117116 25205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } }
I20260620 19:12:53.117493 25206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f919694baf642efb52c582cffc9ccc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } } }
I20260620 19:12:53.117573 25206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:53.117736 25207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f919694baf642efb52c582cffc9ccc2. Latest consensus state: current_term: 1 leader_uuid: "6f919694baf642efb52c582cffc9ccc2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f919694baf642efb52c582cffc9ccc2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42041 } } }
I20260620 19:12:53.117797 25207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2 [sys.catalog]: This master's current role is: LEADER
W20260620 19:12:53.118748 25221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:12:53.118791 25221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:12:53.118848 25212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:53.119006 25212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:53.120311 25212 catalog_manager.cc:1347] Generated new cluster ID: 6c6407cfff88433fb0f89a66837f89c6
I20260620 19:12:53.120358 25212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:53.125769 25212 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:53.126241 25212 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:53.128602 25212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f919694baf642efb52c582cffc9ccc2: Generated new TSK 0
I20260620 19:12:53.128748 25212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:12:53.186846 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:53.139583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.19.66.254:42041)
0620 19:12:53.139637 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:53.139640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:53.139705 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:12:53.139767 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:53.139768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:53.139781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:53.139871 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0620 19:12:53.139955 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.140715 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.141509 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.141847 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.141920 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:53.142038 (+ 118us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 19:12:53.186558 (+ 44520us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:12:53.186768 (+ 210us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42010: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":30}
W20260620 19:12:53.186928 25203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42010: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 19:12:53.186926 25229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:53.139515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:42041 (local address 127.0.0.1:42010)
0620 19:12:53.139659 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:53.139667 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:53.139715 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:53.139869 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:53.139869 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:53.139871 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 19:12:53.139917 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:53.139918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.140814 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.140815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:53.141438 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:53.141441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:53.141916 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:53.141917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:53.141964 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:53.141969 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 19:12:53.186715 (+ 44746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 19:12:53.186835 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.19.66.254:42041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260620 19:12:53.187374 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25133
2026-06-20T19:12:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (216 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (84 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:53 dist-test-slave-nq8r krb5kdc[25241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:53 dist-test-slave-nq8r krb5kdc[25241](info): set up 2 sockets
Jun 20 19:12:53 dist-test-slave-nq8r krb5kdc[25241](info): commencing operation
krb5kdc: starting...
W20260620 19:12:54.296753 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:12:54.399170 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43289
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.476404 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.476608 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.476636 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.477828 25254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:54.477869 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.477887 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.477903 25254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:54.477917 25254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:54.479314 25254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:54.479544 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.479743 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.484251 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.484367 25254 server_base.cc:1034] running on GCE node
W20260620 19:12:54.484258 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.484259 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.484704 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.485044 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.486198 25254 hybrid_clock.cc:648] HybridClock initialized: now 1781982774486181 us; error 27 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.488590 25254 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:12:54.489022 25254 webserver.cc:460] Webserver started at http://127.19.66.254:45829/ using document root <none> and password file <none>
I20260620 19:12:54.489234 25254 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.489280 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.489421 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.490442 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "4d63302fefad408aa043fe93c445ed04"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "4fc5ef30d66d4d365be76533bc67d301"
server_key_iv: "b744139341c75cbd57e8bbfcdda1b50a"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.490769 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "4d63302fefad408aa043fe93c445ed04"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "4fc5ef30d66d4d365be76533bc67d301"
server_key_iv: "b744139341c75cbd57e8bbfcdda1b50a"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.491972 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:54.492609 25269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.492745 25254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:54.492825 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "4d63302fefad408aa043fe93c445ed04"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "4fc5ef30d66d4d365be76533bc67d301"
server_key_iv: "b744139341c75cbd57e8bbfcdda1b50a"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.492915 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.509738 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.511814 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.511951 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.515617 25254 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:43289
I20260620 19:12:54.515667 25321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:43289 every 8 connection(s)
I20260620 19:12:54.515980 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:54.516944 25322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:54.519135 25322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04: Bootstrap starting.
I20260620 19:12:54.519685 25322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:54.519945 25322 log.cc:824] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:54.520614 25322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04: No bootstrap required, opened a new log
I20260620 19:12:54.522161 25322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } }
I20260620 19:12:54.522305 25322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:54.522336 25322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d63302fefad408aa043fe93c445ed04, State: Initialized, Role: FOLLOWER
I20260620 19:12:54.522470 25322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } }
I20260620 19:12:54.522526 25322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:54.522547 25322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:54.522593 25322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:54.523077 25322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } }
I20260620 19:12:54.523105 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25254
I20260620 19:12:54.523185 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:54.523172 25322 leader_election.cc:302] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d63302fefad408aa043fe93c445ed04; no voters:
I20260620 19:12:54.523352 19723 external_mini_cluster.cc:1428] Setting key 65efc51afc47671c71cd4f19964df92b
I20260620 19:12:54.523371 25322 leader_election.cc:288] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:54.523413 25325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:54.523633 25322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:54.523727 25325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [term 1 LEADER]: Becoming Leader. State: Replica: 4d63302fefad408aa043fe93c445ed04, State: Running, Role: LEADER
I20260620 19:12:54.523816 25325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } }
I20260620 19:12:54.524145 25326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d63302fefad408aa043fe93c445ed04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } } }
I20260620 19:12:54.524224 25326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:54.524343 25326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d63302fefad408aa043fe93c445ed04. Latest consensus state: current_term: 1 leader_uuid: "4d63302fefad408aa043fe93c445ed04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d63302fefad408aa043fe93c445ed04" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43289 } } }
I20260620 19:12:54.524420 25326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04 [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:54.524718 25333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:54.525261 25333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:12:54.526721 25333 catalog_manager.cc:1347] Generated new cluster ID: b268791e0ab64ee1a39d1f279eded3ef
I20260620 19:12:54.526772 25333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:54.534551 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.523764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.19.66.254:43289)
0620 19:12:54.524006 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.524015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.524044 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:12:54.524154 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.524162 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.524292 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.525315 (+ 1023us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.525431 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.526732 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.527771 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.528345 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.528439 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.531136 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.531140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.531141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.531153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.533040 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.533386 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.533389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.533389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.533499 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.533665 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.533665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.533666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.533962 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.534118 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.534193 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.534327 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260620 19:12:54.545766 25333 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:54.546470 25333 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:54.553685 25333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d63302fefad408aa043fe93c445ed04: Generated new TSK 0
I20260620 19:12:54.553859 25333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:12:54.577319 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.679564 25348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.679771 25348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.679793 25348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.681015 25348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.681056 25348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.681080 25348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:54.682570 25348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:54.682786 25348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.683012 25348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.687549 25356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.687570 25354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.687660 25348 server_base.cc:1034] running on GCE node
W20260620 19:12:54.687609 25353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.687953 25348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.688151 25348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.689288 25348 hybrid_clock.cc:648] HybridClock initialized: now 1781982774689278 us; error 25 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.691648 25348 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:12:54.692030 25348 webserver.cc:460] Webserver started at http://127.19.66.193:33733/ using document root <none> and password file <none>
I20260620 19:12:54.692216 25348 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.692261 25348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.692385 25348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.693397 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "7bac1c019c084ff7b132b4baa5e798c3"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "c95fd814a47439dd03bce14a7e4ba1af"
server_key_iv: "2530ce1a380dbc5711b8e0255f9596e4"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.693714 25348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "7bac1c019c084ff7b132b4baa5e798c3"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "c95fd814a47439dd03bce14a7e4ba1af"
server_key_iv: "2530ce1a380dbc5711b8e0255f9596e4"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.694928 25348 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:54.695583 25363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.695722 25348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:54.695801 25348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "7bac1c019c084ff7b132b4baa5e798c3"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "c95fd814a47439dd03bce14a7e4ba1af"
server_key_iv: "2530ce1a380dbc5711b8e0255f9596e4"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.695871 25348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.707978 25348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.710032 25348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.710161 25348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.710367 25348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:54.710654 25348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:54.710683 25348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.710711 25348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:54.710734 25348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.716048 25348 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:33447
I20260620 19:12:54.716086 25476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:33447 every 8 connection(s)
I20260620 19:12:54.716391 25348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:54.722033 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25348
I20260620 19:12:54.722124 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:54.722303 19723 external_mini_cluster.cc:1428] Setting key e375f23e8e5e13f72996cb6054618b85
I20260620 19:12:54.725217 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.717056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:38637 (local address 127.19.66.254:43289)
0620 19:12:54.717134 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.717137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.717309 (+ 172us) server_negotiation.cc:407] Connection header received
0620 19:12:54.718169 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.718170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.718182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.718218 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.718770 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.719283 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.720199 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.720429 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.720481 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.722531 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.722533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.722533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.722540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.723871 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.724213 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.724214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.724215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.724292 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.724517 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.724518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.724518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.724680 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.724735 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.725086 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.725156 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:12:54.725409 25479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.717043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43289 (local address 127.19.66.193:38637)
0620 19:12:54.717197 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:54.717232 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:54.718075 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:54.718278 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:54.718282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:54.718518 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:54.718698 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.718702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.719345 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.719346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.720119 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.720122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.720490 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.720490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.720583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.721082 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:54.721090 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:54.722438 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:54.723944 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.723951 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.723953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.724152 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.724341 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.724342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.724342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.724463 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.724762 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:54.724769 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:54.724948 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 19:12:54.725056 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:12:54.725206 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260620 19:12:54.725947 25477 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43289
I20260620 19:12:54.726033 25477 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:54.726203 25477 heartbeater.cc:511] Master 127.19.66.254:43289 requested a full tablet report, sending...
I20260620 19:12:54.726676 25286 ts_manager.cc:194] Registered new tserver with Master: 7bac1c019c084ff7b132b4baa5e798c3 (127.19.66.193:33447)
I20260620 19:12:54.727263 25286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:38637
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:12:54.743268 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.821077 25482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.821269 25482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.821297 25482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.822427 25482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.822463 25482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.822497 25482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:54.823980 25482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:54.824200 25482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.824395 25482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.829059 25490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.829084 25488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.829284 25482 server_base.cc:1034] running on GCE node
W20260620 19:12:54.829059 25487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.829523 25482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.829741 25482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.830870 25482 hybrid_clock.cc:648] HybridClock initialized: now 1781982774830856 us; error 27 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.833079 25482 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:12:54.833475 25482 webserver.cc:460] Webserver started at http://127.19.66.194:42785/ using document root <none> and password file <none>
I20260620 19:12:54.833670 25482 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.833714 25482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.833842 25482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.834815 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "0a621196a8e9487ba82e2feaa909bccb"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "df742dda518a1c3c99dc6a8bba594bc7"
server_key_iv: "ba397497acb16e67ebc41458c29e73ab"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.835125 25482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "0a621196a8e9487ba82e2feaa909bccb"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "df742dda518a1c3c99dc6a8bba594bc7"
server_key_iv: "ba397497acb16e67ebc41458c29e73ab"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.836287 25482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:12:54.836922 25497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.837059 25482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.837136 25482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "0a621196a8e9487ba82e2feaa909bccb"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "df742dda518a1c3c99dc6a8bba594bc7"
server_key_iv: "ba397497acb16e67ebc41458c29e73ab"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.837200 25482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:54.868057 25482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:54.869777 25482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:54.869912 25482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:54.870118 25482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:54.870410 25482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:54.870443 25482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.870472 25482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:54.870489 25482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.875703 25482 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38313
I20260620 19:12:54.875744 25610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38313 every 8 connection(s)
I20260620 19:12:54.876039 25482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:12:54.877462 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25482
I20260620 19:12:54.877542 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:12:54.877710 19723 external_mini_cluster.cc:1428] Setting key f55e07f07ba03616b3f640a1907361ed
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
I20260620 19:12:54.885062 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.876683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:49171 (local address 127.19.66.254:43289)
0620 19:12:54.876764 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:54.876767 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:54.876925 (+ 158us) server_negotiation.cc:407] Connection header received
0620 19:12:54.877858 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:54.877859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:54.877871 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:54.877914 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:54.878550 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.879225 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.880212 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.880417 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.880473 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.882610 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:54.882611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:54.882611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:54.882619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:54.883825 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.884178 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.884179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.884179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.884229 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:54.884439 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:54.884440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:54.884441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:54.884548 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:54.884601 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:54.884875 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 19:12:54.884968 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 19:12:54.885195 25613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:54.876653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43289 (local address 127.19.66.194:49171)
0620 19:12:54.876795 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:54.876828 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:54.877745 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:54.877991 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:54.877995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:54.878236 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:54.878472 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.878477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.879297 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.879297 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.880129 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:54.880131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:54.880485 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:54.880486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:54.880608 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:54.881128 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:54.881139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:54.882527 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:54.883896 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.883903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.883905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.884114 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.884281 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:54.884281 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:54.884282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:54.884378 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:54.884601 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:54.884607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:54.884735 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:12:54.884820 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:12:54.884983 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260620 19:12:54.885664 25611 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43289
I20260620 19:12:54.885749 25611 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:54.885912 25611 heartbeater.cc:511] Master 127.19.66.254:43289 requested a full tablet report, sending...
I20260620 19:12:54.886236 25286 ts_manager.cc:194] Registered new tserver with Master: 0a621196a8e9487ba82e2feaa909bccb (127.19.66.194:38313)
I20260620 19:12:54.886587 25286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:49171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:12:54.897481 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:54.974745 25616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:54.974990 25616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:54.975024 25616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:54.976230 25616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:54.976279 25616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:54.976313 25616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:12:54.978355 25616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:54.978598 25616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:54.978849 25616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:54.983923 25624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.983939 25621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:54.983938 25622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:54.984138 25616 server_base.cc:1034] running on GCE node
I20260620 19:12:54.984362 25616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:54.984598 25616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:54.985746 25616 hybrid_clock.cc:648] HybridClock initialized: now 1781982774985724 us; error 35 us; skew 500 ppm
Jun 20 19:12:54 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:54.987999 25616 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:12:54.988440 25616 webserver.cc:460] Webserver started at http://127.19.66.195:33255/ using document root <none> and password file <none>
I20260620 19:12:54.988643 25616 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:54.988690 25616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:54.988811 25616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:54.989835 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "772c84bd7b894e99aafb23e14cc303ae"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "08a91707f73265bce29b84e068f8c16e"
server_key_iv: "f021e6708b5ae18a5783424747ae3fe3"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.990163 25616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "772c84bd7b894e99aafb23e14cc303ae"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "08a91707f73265bce29b84e068f8c16e"
server_key_iv: "f021e6708b5ae18a5783424747ae3fe3"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.991382 25616 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:54.991979 25631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:54.992110 25616 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:54.992185 25616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "772c84bd7b894e99aafb23e14cc303ae"
format_stamp: "Formatted at 2026-06-20 19:12:54 on dist-test-slave-nq8r"
server_key: "08a91707f73265bce29b84e068f8c16e"
server_key_iv: "f021e6708b5ae18a5783424747ae3fe3"
server_key_version: "encryptionkey@0"
I20260620 19:12:54.992242 25616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:55.012248 25616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:55.014050 25616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:55.014181 25616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:55.014415 25616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:55.014698 25616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:55.014730 25616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:55.014760 25616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:55.014780 25616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:55.020334 25616 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:46349
I20260620 19:12:55.020371 25744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:46349 every 8 connection(s)
I20260620 19:12:55.020674 25616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:12:55.021282 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25616
I20260620 19:12:55.021375 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:12:55.021587 19723 external_mini_cluster.cc:1428] Setting key 22833d2ddd184f96c8b1aeca42d2eb44
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982774, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:12:55.029421 25329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.021348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:58335 (local address 127.19.66.254:43289)
0620 19:12:55.021400 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.021402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.021683 (+ 281us) server_negotiation.cc:407] Connection header received
0620 19:12:55.022671 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.022672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.022683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.022734 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.023301 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.023788 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.024837 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.025128 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.025201 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.027164 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.027165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.027165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.027172 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.028221 (+ 1049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.028560 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.028561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.028561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.028615 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.028824 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.028825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.028825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.028959 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.029013 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.029253 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.029332 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:12:55.029577 25747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.021332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43289 (local address 127.19.66.195:58335)
0620 19:12:55.021570 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:55.021609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:55.022583 (+ 974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:55.022800 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:55.022804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:55.023043 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:55.023222 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.023226 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.023889 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.023890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.024756 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:55.024758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.025197 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.025198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:55.025344 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.025810 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:55.025819 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:55.027084 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:55.028288 (+ 1204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:55.028294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:55.028304 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:55.028498 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:55.028668 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:55.028668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:55.028668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:55.028768 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:55.029012 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:55.029017 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:55.029143 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 19:12:55.029227 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:12:55.029379 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":70,"threads_started":1}
I20260620 19:12:55.029924 25745 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43289
I20260620 19:12:55.030009 25745 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:55.030160 25745 heartbeater.cc:511] Master 127.19.66.254:43289 requested a full tablet report, sending...
I20260620 19:12:55.030454 25286 ts_manager.cc:194] Registered new tserver with Master: 772c84bd7b894e99aafb23e14cc303ae (127.19.66.195:46349)
I20260620 19:12:55.030805 25286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:58335
I20260620 19:12:55.033031 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/test-admin.keytab.
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.045828 19723 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 19:12:55.047292 25752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:55.046605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57168 (local address 127.19.66.193:33447)
0620 19:12:55.046774 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.046779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.046788 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:12:55.046827 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.046829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.046842 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.046903 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.047207 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57168: BlockingRecv error: recv got EOF from 127.0.0.1:57168 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
W20260620 19:12:55.047467 19723 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.048362 19723 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.049156 19723 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:55.055944 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.049619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57170 (local address 127.19.66.193:33447)
0620 19:12:55.049666 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.049668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.049706 (+ 38us) server_negotiation.cc:407] Connection header received
0620 19:12:55.049775 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.049776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.049789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.049837 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.050191 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.050913 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.051578 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.051896 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.051953 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.053805 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.053813 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.053814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.053820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.055096 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.055369 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.055371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.055372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.055420 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.055543 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.055543 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.055543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.055649 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.055737 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.055789 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.055848 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 19:12:55.057520 25752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:55.057030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.19.66.193:33447)
0620 19:12:55.057065 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.057067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.057115 (+ 48us) server_negotiation.cc:407] Connection header received
0620 19:12:55.057178 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.057179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.057189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.057228 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.057480 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57176: BlockingRecv error: recv got EOF from 127.0.0.1:57176 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:12:55.057580 19723 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.058404 19723 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.059108 19723 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:55.064949 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.059519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.19.66.193:33447)
0620 19:12:55.059563 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.059565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.059611 (+ 46us) server_negotiation.cc:407] Connection header received
0620 19:12:55.059648 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.059648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.059655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.059694 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.059976 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.060431 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.061078 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.061266 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.061315 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.063044 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.063045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.063046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.063053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.064040 (+ 987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.064319 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.064320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.064321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.064366 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.064518 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.064519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.064519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.064602 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.064680 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.064772 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.064843 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 19:12:55.066147 25752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:12:55.065793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.19.66.193:33447)
0620 19:12:55.065832 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.065833 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.065844 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:12:55.065885 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.065885 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.065892 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.065930 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.066107 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57198: BlockingRecv error: recv got EOF from 127.0.0.1:57198 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:12:55.066238 19723 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781982751812665-19723-0/krb5kdc/krb5cc' of type FILE
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.067055 19723 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:55.067737 19723 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 19:12:55 dist-test-slave-nq8r krb5kdc[25241](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:12:55.073510 25752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:55.068151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.19.66.193:33447)
0620 19:12:55.068197 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:55.068198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:55.068205 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:12:55.068251 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:55.068251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:55.068258 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:55.068315 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:55.068593 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.069083 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.069697 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:55.069877 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:55.069923 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:55.071609 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:55.071611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:55.071611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:55.071617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:55.072678 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.073008 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.073009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.073009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.073054 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:55.073177 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:55.073177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:55.073177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:55.073250 (+ 73us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:55.073329 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:55.073378 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:12:55.073429 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:12:55.074124 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25348
I20260620 19:12:55.079027 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25482
I20260620 19:12:55.082859 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25616
I20260620 19:12:55.086398 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25254
2026-06-20T19:12:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1732 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:12:56 dist-test-slave-nq8r krb5kdc[25765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:12:56 dist-test-slave-nq8r krb5kdc[25765](info): set up 2 sockets
Jun 20 19:12:56 dist-test-slave-nq8r krb5kdc[25765](info): commencing operation
krb5kdc: starting...
W20260620 19:12:57.745157 19723 mini_kdc.cc:121] Time spent starting KDC: real 2.640s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260620 19:12:57.934917 19723 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.190s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 19:12:58.150286 19723 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.215s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260620 19:12:58.349781 19723 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.199s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/test-user.keytab.
W20260620 19:12:59.064945 19723 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.715s user 0.000s sys 0.001s
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:12:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:12:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab.
Entry for principal oryx/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab.
W20260620 19:12:59.553217 19723 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.19.66.254: real 0.468s user 0.000s sys 0.002s
I20260620 19:12:59.553565 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46419
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab
--principal=oryx/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:59.631127 25778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:59.631330 25778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:59.631352 25778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:59.632488 25778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:12:59.632531 25778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:59.632544 25778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:59.632566 25778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:12:59.632582 25778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:12:59.634040 25778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab
--principal=oryx/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:59.634262 25778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:59.634473 25778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:59.639081 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:59.639219 25784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:59.639235 25786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:59.639093 25778 server_base.cc:1034] running on GCE node
I20260620 19:12:59.639534 25778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:59.639811 25778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:59.640915 25778 hybrid_clock.cc:648] HybridClock initialized: now 1781982779640891 us; error 34 us; skew 500 ppm
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:59.647610 25778 init.cc:377] Logged in from keytab as oryx/127.19.66.254@KRBTEST.COM (short username oryx)
I20260620 19:12:59.648077 25778 webserver.cc:460] Webserver started at http://127.19.66.254:36081/ using document root <none> and password file <none>
I20260620 19:12:59.648285 25778 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:59.648330 25778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:59.648465 25778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:59.649570 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f50f03620cc74558967c7da487a3f08d"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "7685dda5ce67d183a25dbfec9c9d2ba6"
server_key_iv: "750227cf5460ca729fac378558e90c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.649904 25778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f50f03620cc74558967c7da487a3f08d"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "7685dda5ce67d183a25dbfec9c9d2ba6"
server_key_iv: "750227cf5460ca729fac378558e90c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.651117 25778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:59.651791 25793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:59.651942 25778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:59.652024 25778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f50f03620cc74558967c7da487a3f08d"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "7685dda5ce67d183a25dbfec9c9d2ba6"
server_key_iv: "750227cf5460ca729fac378558e90c5f"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.652091 25778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:59.668536 25778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:59.670387 25778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:59.670533 25778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:59.674417 25778 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:46419
I20260620 19:12:59.674445 25845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:46419 every 8 connection(s)
I20260620 19:12:59.674779 25778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:12:59.675688 25846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:12:59.677907 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Bootstrap starting.
I20260620 19:12:59.678032 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25778
I20260620 19:12:59.678133 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:12:59.678297 19723 external_mini_cluster.cc:1428] Setting key 5caff78fe44dfba9887795c6b6b7018c
I20260620 19:12:59.678551 25846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Neither blocks nor log segments found. Creating new log.
I20260620 19:12:59.678860 25846 log.cc:824] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:12:59.679632 25846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: No bootstrap required, opened a new log
I20260620 19:12:59.681888 25846 raft_consensus.cc:348] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } }
I20260620 19:12:59.682065 25846 raft_consensus.cc:374] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:12:59.682099 25846 raft_consensus.cc:729] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50f03620cc74558967c7da487a3f08d, State: Initialized, Role: FOLLOWER
I20260620 19:12:59.682233 25846 consensus_queue.cc:260] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } }
I20260620 19:12:59.682317 25846 raft_consensus.cc:388] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:12:59.682349 25846 raft_consensus.cc:482] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:12:59.682397 25846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:12:59.683081 25846 raft_consensus.cc:504] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } }
I20260620 19:12:59.683190 25846 leader_election.cc:302] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f50f03620cc74558967c7da487a3f08d; no voters:
I20260620 19:12:59.683352 25846 leader_election.cc:288] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:12:59.683424 25851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:12:59.683658 25851 raft_consensus.cc:686] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 LEADER]: Becoming Leader. State: Replica: f50f03620cc74558967c7da487a3f08d, State: Running, Role: LEADER
I20260620 19:12:59.683758 25851 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } }
I20260620 19:12:59.684011 25846 sys_catalog.cc:564] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:12:59.684068 25851 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50f03620cc74558967c7da487a3f08d. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } }
I20260620 19:12:59.684345 25851 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:59.684242 25852 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } }
I20260620 19:12:59.684785 25856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:12:59.684800 25852 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:12:59.685272 25856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:12:59.687144 25856 catalog_manager.cc:1347] Generated new cluster ID: e7e028618ce74e65806a6e824b3e47e0
I20260620 19:12:59.687189 25856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:12:59.691277 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:59.678792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.19.66.254:46419)
0620 19:12:59.678988 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:59.678998 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:59.679034 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:12:59.679573 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:59.679583 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:59.679643 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:59.679793 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:59.680346 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.681814 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.682691 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.683312 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.683426 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:59.686199 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:59.686204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:59.686204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:59.686216 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:59.689761 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:59.690166 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:59.690169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:59.690169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:59.690265 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:59.690448 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:59.690449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:59.690449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:59.690719 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:59.690858 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:59.690969 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 19:12:59.691091 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":105,"threads_started":1}
I20260620 19:12:59.703176 25856 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:12:59.703896 25856 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:12:59.717334 25856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Generated new TSK 0
I20260620 19:12:59.717545 25856 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
I20260620 19:12:59.748972 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:59.829063 25872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:59.829334 25872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:59.829368 25872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:59.831056 25872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:59.831115 25872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:59.831156 25872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:12:59.833261 25872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:59.833521 25872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:59.833784 25872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:59.838869 25877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:59.838861 25880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:59.838968 25872 server_base.cc:1034] running on GCE node
W20260620 19:12:59.839289 25878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:59.839542 25872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:59.839779 25872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:59.840909 25872 hybrid_clock.cc:648] HybridClock initialized: now 1781982779840892 us; error 30 us; skew 500 ppm
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:59.843170 25872 init.cc:377] Logged in from keytab as oryx/127.19.66.193@KRBTEST.COM (short username oryx)
I20260620 19:12:59.843546 25872 webserver.cc:460] Webserver started at http://127.19.66.193:33885/ using document root <none> and password file <none>
I20260620 19:12:59.843713 25872 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:59.843750 25872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:59.843838 25872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:59.844821 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "9c1e6737a4894482a00e28e4baa20e1a"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "f7543a2e8b5f3ff38209e3153cf62f1d"
server_key_iv: "b2aa61f535d70b6ab90a4686503ed326"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.845160 25872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "9c1e6737a4894482a00e28e4baa20e1a"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "f7543a2e8b5f3ff38209e3153cf62f1d"
server_key_iv: "b2aa61f535d70b6ab90a4686503ed326"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.846287 25872 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:12:59.846875 25887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:59.847005 25872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:59.847083 25872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "9c1e6737a4894482a00e28e4baa20e1a"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "f7543a2e8b5f3ff38209e3153cf62f1d"
server_key_iv: "b2aa61f535d70b6ab90a4686503ed326"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.847141 25872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:12:59.861213 25872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:12:59.863235 25872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:12:59.863361 25872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:12:59.863569 25872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:12:59.863842 25872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:12:59.863873 25872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:59.863901 25872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:12:59.863925 25872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:59.869086 25872 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:41777
I20260620 19:12:59.869144 26000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:41777 every 8 connection(s)
I20260620 19:12:59.869398 25872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:12:59.873267 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 25872
I20260620 19:12:59.873359 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:12:59.873545 19723 external_mini_cluster.cc:1428] Setting key dd7e1004a17515d9a823c93f16dc0537
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:12:59.878005 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:59.870038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:44885 (local address 127.19.66.254:46419)
0620 19:12:59.870095 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:12:59.870096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:12:59.870261 (+ 165us) server_negotiation.cc:407] Connection header received
0620 19:12:59.870758 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:12:59.870759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:12:59.870768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:12:59.870807 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:12:59.871649 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.872137 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.873117 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.873304 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.873362 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:59.875427 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:12:59.875428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:12:59.875429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:12:59.875435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:12:59.876648 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:59.877110 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:59.877111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:59.877111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:59.877161 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:12:59.877422 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:12:59.877423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:12:59.877423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:12:59.877533 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:12:59.877581 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:12:59.877877 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 19:12:59.877943 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:12:59.878245 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:12:59.870003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:44885)
0620 19:12:59.870152 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:12:59.870192 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:12:59.870665 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:12:59.870864 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:12:59.870867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:12:59.871386 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:12:59.871582 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:59.871586 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.872196 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.872196 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:59.873055 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:12:59.873058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:12:59.873383 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:12:59.873384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:12:59.873535 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:12:59.873978 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:12:59.873987 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:12:59.875344 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:12:59.876723 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:59.876730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:59.876732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:59.877036 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:59.877228 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:12:59.877229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:12:59.877229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:12:59.877362 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:12:59.877590 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:12:59.877596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:12:59.877753 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:12:59.877867 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:12:59.878026 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260620 19:12:59.878710 26001 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:12:59.878793 26001 heartbeater.cc:464] Registering TS with master...
I20260620 19:12:59.878958 26001 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:12:59.879419 25810 ts_manager.cc:194] Registered new tserver with Master: 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:12:59.879992 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:44885
WARNING: no policy specified for oryx/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
I20260620 19:12:59.894901 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:12:59.971352 26006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:12:59.971546 26006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:12:59.971596 26006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:12:59.972704 26006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:12:59.972744 26006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:12:59.972777 26006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:12:59.974256 26006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:12:59.974478 26006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:12:59.974680 26006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:12:59.979172 26012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:12:59.979183 26011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:59.979408 26006 server_base.cc:1034] running on GCE node
W20260620 19:12:59.979179 26014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:12:59.979664 26006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:12:59.979892 26006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:12:59.981029 26006 hybrid_clock.cc:648] HybridClock initialized: now 1781982779981016 us; error 26 us; skew 500 ppm
Jun 20 19:12:59 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:12:59.983229 26006 init.cc:377] Logged in from keytab as oryx/127.19.66.194@KRBTEST.COM (short username oryx)
I20260620 19:12:59.983669 26006 webserver.cc:460] Webserver started at http://127.19.66.194:41457/ using document root <none> and password file <none>
I20260620 19:12:59.983863 26006 fs_manager.cc:359] Metadata directory not provided
I20260620 19:12:59.983902 26006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:12:59.983994 26006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:12:59.985080 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "2b7dd5b0f71593d0478965b87fbbc34a"
server_key_iv: "482f340a7c599d690473fdc5c81dfb16"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.985401 26006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "2b7dd5b0f71593d0478965b87fbbc34a"
server_key_iv: "482f340a7c599d690473fdc5c81dfb16"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.986654 26006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:12:59.987231 26021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:12:59.987373 26006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:12:59.987447 26006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "2b7dd5b0f71593d0478965b87fbbc34a"
server_key_iv: "482f340a7c599d690473fdc5c81dfb16"
server_key_version: "encryptionkey@0"
I20260620 19:12:59.987514 26006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.010989 26006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.013243 26006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.013379 26006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.013670 26006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:00.013984 26006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:00.014020 26006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.014050 26006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:00.014074 26006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.019570 26006 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:33997
I20260620 19:13:00.019641 26134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:33997 every 8 connection(s)
I20260620 19:13:00.019928 26006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:00.028312 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.020613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:54557 (local address 127.19.66.254:46419)
0620 19:13:00.020674 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.020676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.020866 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:13:00.021377 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.021378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.021387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.021429 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.022297 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.022786 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.023744 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.023921 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.023966 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.026006 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.026008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.026008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.026015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.027131 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.027541 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.027541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.027542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.027590 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.027823 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.027823 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.027823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.027923 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.027969 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.028183 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.028242 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:13:00.028522 26137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.020594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:54557)
0620 19:13:00.020751 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.020788 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.021296 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.021492 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.021496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.022037 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.022232 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.022236 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.022862 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.022863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.023680 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.023683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.023988 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.023988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.024096 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.024604 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.024612 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.025916 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.027216 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.027222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.027224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.027473 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.027660 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.027661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.027661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.027773 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.027965 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.027969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.028084 (+ 115us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.028162 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.028283 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260620 19:13:00.029027 26135 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:13:00.029116 26135 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.029099 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26006
I20260620 19:13:00.029193 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:00.029309 26135 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:13:00.029377 19723 external_mini_cluster.cc:1428] Setting key 0157ff9add3fb9fa6da34f925591e960
I20260620 19:13:00.029698 25810 ts_manager.cc:194] Registered new tserver with Master: 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997)
I20260620 19:13:00.030205 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:54557
WARNING: no policy specified for oryx/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
I20260620 19:13:00.049731 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:00.127401 26140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:00.127607 26140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:00.127628 26140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:00.128746 26140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:00.128790 26140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:00.128815 26140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:00.130229 26140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:00.130435 26140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:00.130642 26140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.135191 26146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.135170 26148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.135249 26140 server_base.cc:1034] running on GCE node
W20260620 19:13:00.135160 26145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.135568 26140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.135779 26140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.136894 26140 hybrid_clock.cc:648] HybridClock initialized: now 1781982780136880 us; error 28 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.139134 26140 init.cc:377] Logged in from keytab as oryx/127.19.66.195@KRBTEST.COM (short username oryx)
I20260620 19:13:00.139564 26140 webserver.cc:460] Webserver started at http://127.19.66.195:37335/ using document root <none> and password file <none>
I20260620 19:13:00.139734 26140 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.139772 26140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.139858 26140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:00.140831 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "3aa0eeb2e8dc409b98093512e408b54c"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "f2a7d6d931cf5af6b43ac46953c464fd"
server_key_iv: "1411b536a5ef7c0fbe4dc9307c69c781"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.141182 26140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "3aa0eeb2e8dc409b98093512e408b54c"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "f2a7d6d931cf5af6b43ac46953c464fd"
server_key_iv: "1411b536a5ef7c0fbe4dc9307c69c781"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.142366 26140 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:00.143049 26155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.143201 26140 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:00.143273 26140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "3aa0eeb2e8dc409b98093512e408b54c"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "f2a7d6d931cf5af6b43ac46953c464fd"
server_key_iv: "1411b536a5ef7c0fbe4dc9307c69c781"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.143342 26140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.160599 26140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.162681 26140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.162807 26140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.163005 26140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:00.163295 26140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:00.163327 26140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.163354 26140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:00.163378 26140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.168483 26140 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:42635
I20260620 19:13:00.168551 26268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:42635 every 8 connection(s)
I20260620 19:13:00.168814 26140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:00.173975 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26140
I20260620 19:13:00.174067 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:00.174250 19723 external_mini_cluster.cc:1428] Setting key d88dfcf31be570dc9e10ee4379ee4ed7
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:00.177110 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.169472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:33515 (local address 127.19.66.254:46419)
0620 19:13:00.169519 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.169522 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.169720 (+ 198us) server_negotiation.cc:407] Connection header received
0620 19:13:00.170175 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.170176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.170186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.170234 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.171095 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.171576 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.172510 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.172683 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.172727 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.174872 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.174874 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.174874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.174880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.175989 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.176345 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.176346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.176346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.176394 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.176601 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.176602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.176602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.176698 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.176741 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.176990 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.177048 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:13:00.177297 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.169452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:33515)
0620 19:13:00.169607 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.169643 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.170102 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.170290 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.170293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.170835 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.171029 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.171033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.171642 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.171643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.172446 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.172448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.172734 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.172734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.172823 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.173267 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.173275 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.174793 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.176069 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.176074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.176076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.176279 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.176464 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.176465 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.176465 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.176557 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.176739 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.176741 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.176874 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.176972 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.177093 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 19:13:00.177708 26269 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:13:00.177786 26269 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.177942 26269 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:13:00.178264 25810 ts_manager.cc:194] Registered new tserver with Master: 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195:42635)
I20260620 19:13:00.178596 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:33515
I20260620 19:13:00.185875 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.66.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.253.keytab.
Entry for principal oryx/127.19.66.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.253.keytab.
I20260620 19:13:00.207413 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:43823
--webserver_interface=127.19.66.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.253.keytab
--principal=oryx/127.19.66.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46419,127.19.66.253:43823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:00.286321 26274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:00.286530 26274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:00.286559 26274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:00.287676 26274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:00.287719 26274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:00.287737 26274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:00.287752 26274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:00.287768 26274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:00.289225 26274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46419,127.19.66.253:43823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.253.keytab
--principal=oryx/127.19.66.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:43823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.66.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
W20260620 19:13:00.289333 26274 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.19.66.254:46419,127.19.66.253:43823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 19:13:00.289490 26274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:00.289680 26274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.294188 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.294251 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.294310 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.294472 26274 server_base.cc:1034] running on GCE node
I20260620 19:13:00.294675 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.294950 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.296087 26274 hybrid_clock.cc:648] HybridClock initialized: now 1781982780296072 us; error 27 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.298447 26274 init.cc:377] Logged in from keytab as oryx/127.19.66.253@KRBTEST.COM (short username oryx)
I20260620 19:13:00.298836 26274 webserver.cc:460] Webserver started at http://127.19.66.253:40431/ using document root <none> and password file <none>
I20260620 19:13:00.299041 26274 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.299088 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.299211 26274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:00.300208 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/instance:
uuid: "10deb988d4764b8997e2005e3543510f"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "14b2276798904b507901c1c63d37caf0"
server_key_iv: "741dda66381d37af9e0e5444f2b234c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.300529 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal/instance:
uuid: "10deb988d4764b8997e2005e3543510f"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "14b2276798904b507901c1c63d37caf0"
server_key_iv: "741dda66381d37af9e0e5444f2b234c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.301776 26274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:13:00.302408 26289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.302547 26274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:13:00.302624 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "10deb988d4764b8997e2005e3543510f"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "14b2276798904b507901c1c63d37caf0"
server_key_iv: "741dda66381d37af9e0e5444f2b234c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.302680 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.310753 26274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.312415 26274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.312536 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.316131 26274 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.253:43823
I20260620 19:13:00.316162 26341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.253:43823 every 8 connection(s)
I20260620 19:13:00.316493 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
I20260620 19:13:00.317440 26342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:00.318850 26342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:13:00.321347 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26274
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.253@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
I20260620 19:13:00.327255 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.319449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.19.66.254:46419)
0620 19:13:00.319494 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.319496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.319678 (+ 182us) server_negotiation.cc:407] Connection header received
0620 19:13:00.320114 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.320115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.320124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.320165 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.320735 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.321242 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.322602 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.322783 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.322841 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.324779 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.324780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.324781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.324789 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.326026 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.326392 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.326393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.326393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.326444 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.326646 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.326647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.326647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.326784 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.326838 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.327090 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.327168 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:13:00.327399 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.319414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:46952)
0620 19:13:00.319571 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.319612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.320023 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.320229 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.320232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.320461 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.320660 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.320664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.321318 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.321319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.322523 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.322527 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.322861 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.322862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.322987 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.323464 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.323471 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.324699 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.326096 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.326102 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.326104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.326302 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.326495 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.326496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.326496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.326595 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.326846 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.326852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.326975 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.327060 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.327196 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260620 19:13:00.327973 26342 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:13:00.329192 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.321796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.19.66.253:43823)
0620 19:13:00.321926 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.321930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.321973 (+ 43us) server_negotiation.cc:407] Connection header received
0620 19:13:00.322046 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.322047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.322063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.322118 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.322757 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.323543 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.324218 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.324556 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.324616 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.326687 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.326695 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.326695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.326701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.328155 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.328499 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.328503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.328504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.328559 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.328712 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.328713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.328713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.328882 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.329022 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.329075 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.329130 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":83,"threads_started":1}
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.253@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
I20260620 19:13:00.334241 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.328200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49278)
0620 19:13:00.328240 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.328249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.328281 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.328484 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.328485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.328627 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.328680 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.328681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.329525 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.329526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.330030 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.330032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.330355 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.330356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.330411 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.330687 (+ 276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.330690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.331663 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.333418 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.333419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.333419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.333561 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.333710 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.333711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.333711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.333760 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.333950 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.333950 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.333984 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.334041 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.334126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:13:00.334317 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.328236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.19.66.253:43823)
0620 19:13:00.328378 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.328379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.328386 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:00.328403 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.328403 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.328413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.328475 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.328742 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.329464 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.330093 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.330300 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.330359 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.331730 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.331731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.331732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.331737 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.333352 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.333614 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.333615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.333615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.333663 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.333807 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.333807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.333808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.333901 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.333952 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.334064 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.334173 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260620 19:13:00.335518 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Bootstrap starting.
I20260620 19:13:00.336058 26342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:00.336258 26342 log.cc:824] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:00.337064 26342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: No bootstrap required, opened a new log
I20260620 19:13:00.338398 26342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } }
I20260620 19:13:00.338522 26342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:00.338543 26342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10deb988d4764b8997e2005e3543510f, State: Initialized, Role: FOLLOWER
I20260620 19:13:00.338634 26342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } }
I20260620 19:13:00.338857 26342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:00.338821 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } } }
I20260620 19:13:00.338896 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:13:00.340265 26360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:13:00.340318 26360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:13:00.341776 26274 master_runner.cc:428] Detected that this master 10deb988d4764b8997e2005e3543510f is joining an existing cluster
I20260620 19:13:00.341813 26274 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.66.253:43823
I20260620 19:13:00.345878 26349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:00.345991 26349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } }
I20260620 19:13:00.346186 26349 leader_election.cc:288] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419)
I20260620 19:13:00.346465 25820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10deb988d4764b8997e2005e3543510f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50f03620cc74558967c7da487a3f08d" is_pre_election: true
I20260620 19:13:00.346798 26293 leader_election.cc:302] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 10deb988d4764b8997e2005e3543510f; no voters: f50f03620cc74558967c7da487a3f08d
I20260620 19:13:00.347399 26349 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 19:13:00.348773 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.342673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49280)
0620 19:13:00.342968 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.342975 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.343022 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.343144 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.343145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.343286 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.343329 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.343330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.343925 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.343926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.344658 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.344659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.345180 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.345181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.345241 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.345698 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.345701 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.346111 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.347811 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.347812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.347813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.347996 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.348166 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.348166 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.348167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.348212 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.348466 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.348466 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.348499 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.348572 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.348665 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":87,"threads_started":1}
I20260620 19:13:00.349066 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.342840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.19.66.253:43823)
0620 19:13:00.342874 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.342876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.343006 (+ 130us) server_negotiation.cc:407] Connection header received
0620 19:13:00.343086 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.343087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.343096 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.343143 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.343403 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.343866 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.344736 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.345091 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.345166 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.346194 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.346195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.346196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.346201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.347730 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.348066 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.348067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.348067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.348111 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.348268 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.348269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.348269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.348379 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.348417 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.348595 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.348726 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":18}
I20260620 19:13:00.350416 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.342696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.19.66.254:46419)
0620 19:13:00.342734 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.342736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.342870 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:13:00.342946 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.342947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.342959 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.343020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.343391 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.344149 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.345076 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.345346 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.345404 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.346956 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.346957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.346958 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.346965 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.349387 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.349763 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.349765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.349765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.349819 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.349972 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.349973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.349973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.350077 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.350129 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.350264 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.350330 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:13:00.350430 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.342666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:46964)
0620 19:13:00.342811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.342820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.342893 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.343030 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.343031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.343234 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.343316 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.343318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.344224 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.344225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.345012 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.345014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.345453 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.345454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.345516 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.345858 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.345863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.346263 (+ 400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.349550 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.349550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.349550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.349698 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.349870 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.349871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.349871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.349923 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.350127 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.350128 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.350166 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.350237 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.350330 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":97,"threads_started":1}
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.254@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
I20260620 19:13:00.358827 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.352262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.19.66.253:43823)
0620 19:13:00.352292 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.352294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.352430 (+ 136us) server_negotiation.cc:407] Connection header received
0620 19:13:00.352504 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.352505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.352515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.352559 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.353059 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.353496 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.354207 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.354397 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.354453 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.356554 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.356555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.356556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.356562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.357786 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.358196 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.358197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.358198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.358249 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.358429 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.358429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.358429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.358518 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.358562 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.358722 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.358774 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:13:00.358932 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.352228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49292)
0620 19:13:00.352341 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.352378 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.352442 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.352594 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.352598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.352936 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.352995 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.352997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.353558 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.353559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.354138 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.354140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.354452 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.354453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.354528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.354923 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.354927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.356457 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.357854 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.357860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.357862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.358110 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.358312 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.358312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.358313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.358376 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.358568 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.358574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.358624 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.358707 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.358821 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260620 19:13:00.359462 25810 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.66.253:43823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "10deb988d4764b8997e2005e3543510f"
member_type: NON_VOTER
last_known_addr {
host: "127.19.66.253"
port: 43823
}
attrs {
promote: true
}
}
dest_uuid: "f50f03620cc74558967c7da487a3f08d"
cas_config_opid_index: -1
I20260620 19:13:00.359740 25810 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } }
I20260620 19:13:00.360167 26370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 10deb988d4764b8997e2005e3543510f (127.19.66.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.360445 26316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:00.361228 26316 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 19:13:00.361536 26369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:00.362182 26370 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.66.253:43823
I20260620 19:13:00.362376 26369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.362466 26274 master.cc:537] Master@127.19.66.253:43823 shutting down...
I20260620 19:13:00.362519 26369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:00.363795 26316 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 10deb988d4764b8997e2005e3543510f (127.19.66.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.364764 26378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.364908 26378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.364969 26378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.365021 26349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50f03620cc74558967c7da487a3f08d. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.365083 26349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.365024 26378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.365090 26372 mvcc.cc:205] Tried to move back new op lower bound from 7299001465720512512 to 7299001465584963584. Current Snapshot: MvccSnapshot[applied={T|T < 7299001465720512512 or (T in {7299001465720512512})}]
I20260620 19:13:00.366339 26274 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Raft consensus shutting down.
I20260620 19:13:00.366446 26274 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Raft consensus is shut down!
I20260620 19:13:00.366483 26274 tablet_replica.cc:330] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: stopping tablet replica
I20260620 19:13:00.379026 26274 master.cc:552] Master@127.19.66.253:43823 shutdown complete.
I20260620 19:13:00.379073 26274 master_runner.cc:305] Clearing existing system tablet
I20260620 19:13:00.379180 26274 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.379223 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.379770 26274 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:00.380257 26380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.380411 26274 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.380463 26274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.380481 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "10deb988d4764b8997e2005e3543510f"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "14b2276798904b507901c1c63d37caf0"
server_key_iv: "741dda66381d37af9e0e5444f2b234c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.381107 26274 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:00.382257 26274 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:13:00.382310 26274 log.cc:1196] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 19:13:00.382519 26274 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Deleting consensus metadata
I20260620 19:13:00.382616 26274 master_runner.cc:315] Copying system tablet from 127.19.66.254:46419
I20260620 19:13:00.383140 26274 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.254:46419
I20260620 19:13:00.388909 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.383504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.19.66.254:46419)
0620 19:13:00.383543 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.383545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.383592 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:13:00.383659 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.383660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.383669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.383715 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.384028 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.384656 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.385519 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.385726 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.385780 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.386646 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.386648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.386648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.386656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.387914 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.388208 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.388209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.388210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.388265 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.388428 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.388428 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.388429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.388535 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.388589 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.388722 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.388790 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:00.388929 26386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.383415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:46970)
0620 19:13:00.383528 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.383539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.383609 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.383718 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.383719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.383898 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.383956 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.383958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.384718 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.384720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.385451 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.385454 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.385783 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.385784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.385847 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.386278 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.386283 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.386566 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.387986 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.387987 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.387987 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.388149 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.388315 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.388316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.388316 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.388372 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.388586 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.388587 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.388632 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.388691 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.388792 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":70,"threads_started":1}
I20260620 19:13:00.389178 25830 tablet_copy_service.cc:140] P f50f03620cc74558967c7da487a3f08d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 10deb988d4764b8997e2005e3543510f ({username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:46970)
I20260620 19:13:00.389240 25830 tablet_copy_service.cc:161] P f50f03620cc74558967c7da487a3f08d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 10deb988d4764b8997e2005e3543510f at {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:46970: session id = 10deb988d4764b8997e2005e3543510f-00000000000000000000000000000000
I20260620 19:13:00.389847 25830 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:13:00.390436 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:13:00.391256 26274 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: tablet copy: Starting download of 0 data blocks...
I20260620 19:13:00.391374 26274 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: tablet copy: Starting download of 1 WAL segments...
I20260620 19:13:00.392179 26274 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:13:00.392982 25830 tablet_copy_service.cc:342] P f50f03620cc74558967c7da487a3f08d: Request end of tablet copy session 10deb988d4764b8997e2005e3543510f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:46970
I20260620 19:13:00.393035 25830 tablet_copy_service.cc:433] P f50f03620cc74558967c7da487a3f08d: ending tablet copy session 10deb988d4764b8997e2005e3543510f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 10deb988d4764b8997e2005e3543510f
I20260620 19:13:00.395596 26274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.396582 26387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.396675 26390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.396704 26388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.396960 26274 server_base.cc:1034] running on GCE node
I20260620 19:13:00.397037 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.397161 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.398290 26274 hybrid_clock.cc:648] HybridClock initialized: now 1781982780398280 us; error 26 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.399562 26274 init.cc:377] Logged in from keytab as oryx/127.19.66.253@KRBTEST.COM (short username oryx)
I20260620 19:13:00.399797 26274 webserver.cc:460] Webserver started at http://127.19.66.253:33077/ using document root <none> and password file <none>
I20260620 19:13:00.399902 26274 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.399933 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.400558 26274 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:00.401086 26397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.401188 26274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:13:00.401228 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "10deb988d4764b8997e2005e3543510f"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "14b2276798904b507901c1c63d37caf0"
server_key_iv: "741dda66381d37af9e0e5444f2b234c3"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.401278 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.422755 26274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.424254 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.427100 26449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.253:43823 every 8 connection(s)
I20260620 19:13:00.427081 26274 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.253:43823
I20260620 19:13:00.427423 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
I20260620 19:13:00.428254 26450 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:13:00.428812 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Bootstrap starting.
I20260620 19:13:00.431237 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:00.431483 26450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Bootstrap complete.
I20260620 19:13:00.431696 26450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } }
I20260620 19:13:00.431783 26450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 10deb988d4764b8997e2005e3543510f, State: Initialized, Role: LEARNER
I20260620 19:13:00.431859 26450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } }
I20260620 19:13:00.432054 26450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:00.432024 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } }
I20260620 19:13:00.432206 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.433697 26465 catalog_manager.cc:1259] Loaded cluster ID: e7e028618ce74e65806a6e824b3e47e0
I20260620 19:13:00.434638 26465 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: loading cluster ID for follower catalog manager: success
I20260620 19:13:00.435899 26465 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: acquiring CA information for follower catalog manager: success
I20260620 19:13:00.436260 26465 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:13:00.437496 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.430601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49318 (local address 127.19.66.253:43823)
0620 19:13:00.430730 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.430732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.430738 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:13:00.430756 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.430756 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.430764 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.430822 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.431294 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.432164 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.433036 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.433334 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.433400 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.434490 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.434492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.434492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.434499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.436282 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.436699 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.436700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.436700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.436761 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.436968 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.436969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.436969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.437059 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.437143 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.437289 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.437372 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.19.66.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.252.keytab.
Entry for principal oryx/127.19.66.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.252.keytab.
I20260620 19:13:00.459280 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:45677
--webserver_interface=127.19.66.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.252.keytab
--principal=oryx/127.19.66.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:00.538378 26468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:00.538582 26468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:00.538604 26468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:00.539695 26468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:00.539729 26468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:00.539742 26468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:00.539753 26468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:00.539772 26468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:00.541260 26468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.252.keytab
--principal=oryx/127.19.66.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.66.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:00.541455 26468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:00.541625 26468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.546016 26474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.546033 26476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.546113 26468 server_base.cc:1034] running on GCE node
W20260620 19:13:00.546010 26473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.546443 26468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.546737 26468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.547873 26468 hybrid_clock.cc:648] HybridClock initialized: now 1781982780547859 us; error 29 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.550231 26468 init.cc:377] Logged in from keytab as oryx/127.19.66.252@KRBTEST.COM (short username oryx)
I20260620 19:13:00.550642 26468 webserver.cc:460] Webserver started at http://127.19.66.252:40307/ using document root <none> and password file <none>
I20260620 19:13:00.550868 26468 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.550917 26468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.551033 26468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:00.552042 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/instance:
uuid: "bb4b8e468ccc48628004f01add882a76"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "10a007c5485d4555bf37109be0382234"
server_key_iv: "cb3c03b4f1c87bf58c964694659abc88"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.552363 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal/instance:
uuid: "bb4b8e468ccc48628004f01add882a76"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "10a007c5485d4555bf37109be0382234"
server_key_iv: "cb3c03b4f1c87bf58c964694659abc88"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.553618 26468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 19:13:00.554240 26483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.554415 26468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.554482 26468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "bb4b8e468ccc48628004f01add882a76"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "10a007c5485d4555bf37109be0382234"
server_key_iv: "cb3c03b4f1c87bf58c964694659abc88"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.554550 26468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.566660 26468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.568313 26468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.568415 26468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.571959 26468 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.252:45677
I20260620 19:13:00.571998 26535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.252:45677 every 8 connection(s)
I20260620 19:13:00.572289 26468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
I20260620 19:13:00.573235 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:00.573527 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26468
I20260620 19:13:00.575553 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.252@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:00.582832 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.576245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.19.66.254:46419)
0620 19:13:00.576287 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.576290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.576419 (+ 129us) server_negotiation.cc:407] Connection header received
0620 19:13:00.576441 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.576442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.576451 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.576497 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.576991 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.577475 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.578231 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.578486 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.578551 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.580612 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.580613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.580614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.580622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.581765 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.582087 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.582088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.582089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.582138 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.582325 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.582326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.582326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.582434 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.582490 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.582664 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.582741 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:00.583029 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.576089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:46976)
0620 19:13:00.576228 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.576266 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.576358 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.576525 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.576527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.576820 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.576916 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.576918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.577562 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.577563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.578156 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.578158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.578582 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.578583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.578633 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.578994 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.579003 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.580532 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.581835 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.581841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.581843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.582023 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.582190 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.582191 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.582191 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.582272 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.582488 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.582494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.582561 (+ 67us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.582639 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.582796 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260620 19:13:00.583031 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.574130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47354 (local address 127.19.66.252:45677)
0620 19:13:00.574289 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.574296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.574321 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:13:00.574747 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.574753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.574799 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.574910 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.575615 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.576664 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.577392 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.577798 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.577890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.580151 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.580156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.580157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.580163 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.581782 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.582098 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.582101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.582101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.582174 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.582341 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.582342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.582343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.582539 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.582659 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.582734 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.582860 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260620 19:13:00.583539 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.252@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
I20260620 19:13:00.589699 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.583934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.19.66.253:43823)
0620 19:13:00.583980 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.583982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.583989 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:00.584010 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.584011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.584021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.584066 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.584299 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.584896 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.585556 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.585804 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.585866 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.587359 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.587360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.587360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.587367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.588700 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.589046 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.589048 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.589048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.589097 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.589254 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.589255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.589255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.589358 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.589420 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.589541 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.589613 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:13:00.589735 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.583844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49334)
0620 19:13:00.583897 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.583907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.583939 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.584072 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.584073 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.584193 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.584235 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.584236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.584983 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.584984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.585488 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.585490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.585890 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.585891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.585958 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.586260 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.586264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.587285 (+ 1021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.588775 (+ 1490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.588776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.588777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.588986 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.589151 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.589151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.589151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.589206 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.589412 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.589413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.589453 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.589513 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.589600 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 19:13:00.590082 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.252@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
I20260620 19:13:00.595041 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.590296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47360)
0620 19:13:00.590335 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.590344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.590375 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.590464 (+ 89us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.590464 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.590559 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.590583 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.590584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.591160 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.591161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.591625 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.591627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.591910 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.591910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.591944 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.592224 (+ 280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.592227 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.593250 (+ 1023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.594347 (+ 1097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.594348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.594348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.594492 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.594621 (+ 129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.594621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.594622 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.594664 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.594807 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.594808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.594836 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.594878 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.594956 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:13:00.595085 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.590350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.19.66.252:45677)
0620 19:13:00.590383 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.590385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.590394 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:00.590411 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.590411 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.590418 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.590460 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.590628 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.591102 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.591680 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.591859 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.591902 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.593323 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.593324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.593324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.593329 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.594287 (+ 958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.594543 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.594544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.594544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.594582 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.594695 (+ 113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.594696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.594696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.594771 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.594809 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.594894 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.594992 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":16}
I20260620 19:13:00.596357 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Bootstrap starting.
I20260620 19:13:00.597159 26536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:00.597398 26536 log.cc:824] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:00.598068 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: No bootstrap required, opened a new log
I20260620 19:13:00.599367 26536 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } }
I20260620 19:13:00.599486 26536 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:00.599507 26536 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4b8e468ccc48628004f01add882a76, State: Initialized, Role: FOLLOWER
I20260620 19:13:00.599628 26536 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } }
I20260620 19:13:00.599867 26541 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } } }
I20260620 19:13:00.599918 26536 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:00.599942 26541 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:13:00.601406 26552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:13:00.601459 26552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:13:00.603466 26468 master_runner.cc:428] Detected that this master bb4b8e468ccc48628004f01add882a76 is joining an existing cluster
I20260620 19:13:00.603505 26468 master_runner.cc:432] Initiating AddMaster RPC to add 127.19.66.252:45677
I20260620 19:13:00.611101 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.19.66.253:43823)
0620 19:13:00.604480 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.604482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.604738 (+ 256us) server_negotiation.cc:407] Connection header received
0620 19:13:00.604787 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.604788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.604799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.604863 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.605346 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.606082 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.607146 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.607323 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.607376 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.608288 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.608289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.608289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.608297 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.609870 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.610245 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.610246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.610246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.610301 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.610492 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.610493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.610493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.610614 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.610655 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.610926 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.611005 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:13:00.611635 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49344)
0620 19:13:00.604661 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.604671 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.604736 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.604909 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.604910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.605172 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.605245 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.605247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.606155 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.606156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.607072 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.607074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.607398 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.607399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.607465 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.607897 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.607902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.608205 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.609958 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.609959 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.609960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.610159 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.610369 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.610370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.610370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.610427 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.610699 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.610700 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.610809 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.611439 (+ 630us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.611537 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":49,"thread_start_us":72,"threads_started":1}
I20260620 19:13:00.611853 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.19.66.254:46419)
0620 19:13:00.604442 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.604444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.604799 (+ 355us) server_negotiation.cc:407] Connection header received
0620 19:13:00.604915 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.604916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.604926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.604978 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.605315 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.606077 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.608144 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.608402 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.608463 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.609293 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.609294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.609295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.609302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.610776 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.611138 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.611139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.611139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.611191 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.611370 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.611370 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.611370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.611486 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.611524 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.611693 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.611763 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:13:00.612177 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:46992)
0620 19:13:00.604740 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.604749 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.604822 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.604983 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.604984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.605205 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.605250 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.605252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.607170 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.607171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.608058 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.608061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.608466 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.608467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.608528 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.608925 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.608929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.609230 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.610850 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.610851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.610851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.611060 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.611251 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.611252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.611252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.611311 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.611562 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.611563 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.611598 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.611986 (+ 388us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.612075 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260620 19:13:00.614791 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47370)
0620 19:13:00.609338 (+ 4800us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.609346 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.609405 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.609556 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.609557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.609725 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.609778 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.609779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.610606 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.610607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.611382 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.611383 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.611803 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.611803 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.611860 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.612229 (+ 369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.612234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.612568 (+ 334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.613957 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.613958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.613958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.614101 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.614254 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.614254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.614254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.614304 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.614495 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.614496 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.614530 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.614588 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.614678 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4782,"thread_start_us":67,"threads_started":1}
I20260620 19:13:00.614811 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.604582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.19.66.252:45677)
0620 19:13:00.604619 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.604621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.609387 (+ 4766us) server_negotiation.cc:407] Connection header received
0620 19:13:00.609473 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.609473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.609482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.609508 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.609851 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.610522 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.611461 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.611713 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.611746 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.612638 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.612639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.612639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.612644 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.613862 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.614156 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.614157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.614157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.614203 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.614351 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.614351 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.614352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.614445 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.614496 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.614612 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.614726 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":21}
I20260620 19:13:00.634363 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25872
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982779, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.254@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
I20260620 19:13:00.640443 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:41777
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:46419
--webserver_port=33885
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:00.642581 26538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.634250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.19.66.252:45677)
0620 19:13:00.634283 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.634285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.634322 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:13:00.634406 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.634407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.634418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.634469 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.635399 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.636160 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.637147 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.637450 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.637850 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.639674 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.639676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.639676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.639684 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.641457 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.641838 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.641840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.641840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.641907 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.642101 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.642102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.642103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.642226 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.642293 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.642442 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.642517 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:13:00.642634 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.634216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47378)
0620 19:13:00.634264 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.634276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.634338 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.635069 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.635070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.635253 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.635316 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.635318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.636249 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.636250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.637067 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.637070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.637560 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.637562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.637667 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.638112 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.638117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.639578 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.641553 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.641554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.641555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.641765 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.641981 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.641982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.641983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.642044 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.642288 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.642289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.642339 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.642410 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.642516 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:13:00.643285 25810 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.19.66.252:45677: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bb4b8e468ccc48628004f01add882a76"
member_type: NON_VOTER
last_known_addr {
host: "127.19.66.252"
port: 45677
}
attrs {
promote: true
}
}
dest_uuid: "f50f03620cc74558967c7da487a3f08d"
cas_config_opid_index: 5
I20260620 19:13:00.643533 25810 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } }
I20260620 19:13:00.643867 26370 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb4b8e468ccc48628004f01add882a76 (127.19.66.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.644870 26370 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.19.66.252:45677
I20260620 19:13:00.644945 26370 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.645010 26370 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:00.645298 26510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:00.646628 26510 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 19:13:00.646955 26468 master.cc:537] Master@127.19.66.252:45677 shutting down...
I20260620 19:13:00.648531 26369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:00.648739 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.643964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49356)
0620 19:13:00.644072 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.644082 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.644113 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.645368 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.645369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.645370 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:00.645408 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.645409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.646349 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.646350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.647616 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.647619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.648496 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.648497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.648569 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.648588 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.648633 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.648670 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260620 19:13:00.648952 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.645141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.19.66.253:43823)
0620 19:13:00.645186 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.645188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.645197 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:00.645222 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.645222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.645246 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.645310 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:00.645475 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.646260 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.647717 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.648319 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.648385 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.648464 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.648810 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.648887 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:13:00.649188 26424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260620 19:13:00.650027 25794 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260620 19:13:00.650483 26369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 19:13:00.650938 26424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER bb4b8e468ccc48628004f01add882a76 (127.19.66.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.651552 26468 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:00.651636 26468 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:00.651674 26468 tablet_replica.cc:330] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: stopping tablet replica
I20260620 19:13:00.651924 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50f03620cc74558967c7da487a3f08d. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.652006 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.652034 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: NON_VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: true } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.652071 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.652405 26369 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: attempting to promote NON_VOTER 10deb988d4764b8997e2005e3543510f to VOTER
I20260620 19:13:00.652582 26369 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } }
I20260620 19:13:00.653240 26424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 LEARNER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260620 19:13:00.653474 26487 connection.cc:441] server connection from 127.0.0.1:47378 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47378 (request call id 3) could send its response
W20260620 19:13:00.653568 26487 connection.cc:441] server connection from 127.0.0.1:47378 torn down before Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47378 (request call id 4) could send its response
I20260620 19:13:00.653959 26369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 19:13:00.654731 26424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 10deb988d4764b8997e2005e3543510f (127.19.66.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.655561 26454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.655648 26454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:13:00.654582 26565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 10deb988d4764b8997e2005e3543510f (127.19.66.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.656759 26369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.656891 26369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:00.656985 26369 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.657038 26369 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:00.661217 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.661085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47392)
0620 19:13:00.661139 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.661187 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:13:00.661300 25794 consensus_peers.cc:487] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d -> Peer bb4b8e468ccc48628004f01add882a76 (127.19.66.252:45677): Couldn't send request to peer bb4b8e468ccc48628004f01add882a76. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45677: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:13:00.664924 26468 master.cc:552] Master@127.19.66.252:45677 shutdown complete.
I20260620 19:13:00.664966 26468 master_runner.cc:305] Clearing existing system tablet
I20260620 19:13:00.665081 26468 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.665124 26468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.665791 26468 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:00.666263 26567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.666405 26468 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.666468 26468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.666489 26468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "bb4b8e468ccc48628004f01add882a76"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "10a007c5485d4555bf37109be0382234"
server_key_iv: "cb3c03b4f1c87bf58c964694659abc88"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.667147 26468 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:00.668421 26468 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 19:13:00.668489 26468 log.cc:1196] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 19:13:00.668653 26468 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Deleting consensus metadata
I20260620 19:13:00.668754 26468 master_runner.cc:315] Copying system tablet from 127.19.66.254:46419
I20260620 19:13:00.669421 26468 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: tablet copy: Beginning tablet copy session from remote peer at address 127.19.66.254:46419
I20260620 19:13:00.678599 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.669740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.19.66.254:46419)
0620 19:13:00.669779 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.669781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.671884 (+ 2103us) server_negotiation.cc:407] Connection header received
0620 19:13:00.671919 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.671920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.671933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.671998 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.672313 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.673099 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.674098 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.674407 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.674502 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.675677 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.675678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.675679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.675686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.677358 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.677796 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.677797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.677798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.677862 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.678069 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.678070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.678070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.678199 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.678258 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.678441 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.678518 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:00.678607 26573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.669695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47004)
0620 19:13:00.671784 (+ 2089us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.671796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.671870 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.672001 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.672002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.672193 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.672240 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.672242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.673198 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.673199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.674004 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.674007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.674565 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.674566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.674628 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.675163 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.675168 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.675560 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.677482 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.677483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.677483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.677703 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.677934 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.677935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.677935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.678006 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.678266 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.678267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.678328 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.678406 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.678513 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":64,"threads_started":1}
I20260620 19:13:00.679250 25830 tablet_copy_service.cc:140] P f50f03620cc74558967c7da487a3f08d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bb4b8e468ccc48628004f01add882a76 ({username='oryx', principal='oryx/127.19.66.252@KRBTEST.COM'} at 127.0.0.1:47004)
I20260620 19:13:00.679307 25830 tablet_copy_service.cc:161] P f50f03620cc74558967c7da487a3f08d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bb4b8e468ccc48628004f01add882a76 at {username='oryx', principal='oryx/127.19.66.252@KRBTEST.COM'} at 127.0.0.1:47004: session id = bb4b8e468ccc48628004f01add882a76-00000000000000000000000000000000
I20260620 19:13:00.679836 25830 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 19:13:00.680281 26468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 19:13:00.681522 26468 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: tablet copy: Starting download of 0 data blocks...
I20260620 19:13:00.681653 26468 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: tablet copy: Starting download of 1 WAL segments...
I20260620 19:13:00.682354 26468 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 19:13:00.683296 25830 tablet_copy_service.cc:342] P f50f03620cc74558967c7da487a3f08d: Request end of tablet copy session bb4b8e468ccc48628004f01add882a76-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.19.66.252@KRBTEST.COM'} at 127.0.0.1:47004
I20260620 19:13:00.683350 25830 tablet_copy_service.cc:433] P f50f03620cc74558967c7da487a3f08d: ending tablet copy session bb4b8e468ccc48628004f01add882a76-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bb4b8e468ccc48628004f01add882a76
I20260620 19:13:00.686406 26468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.687494 26574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.689589 26575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.689848 26468 server_base.cc:1034] running on GCE node
W20260620 19:13:00.693581 26577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.693743 26468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.693882 26468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.694998 26468 hybrid_clock.cc:648] HybridClock initialized: now 1781982780694976 us; error 36 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.696466 26468 init.cc:377] Logged in from keytab as oryx/127.19.66.252@KRBTEST.COM (short username oryx)
I20260620 19:13:00.696733 26468 webserver.cc:460] Webserver started at http://127.19.66.252:44731/ using document root <none> and password file <none>
I20260620 19:13:00.696873 26468 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.696913 26468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.697646 26468 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:00.698208 26584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.698345 26468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:00.698395 26468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "bb4b8e468ccc48628004f01add882a76"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "10a007c5485d4555bf37109be0382234"
server_key_iv: "cb3c03b4f1c87bf58c964694659abc88"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.698450 26468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.717402 26468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.719910 26468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.723049 26468 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.252:45677
I20260620 19:13:00.723088 26636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.252:45677 every 8 connection(s)
I20260620 19:13:00.723517 26468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
I20260620 19:13:00.724478 26637 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:13:00.725028 26637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Bootstrap starting.
I20260620 19:13:00.727804 26637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:00.727998 26637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: Bootstrap complete.
I20260620 19:13:00.728168 26637 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } }
I20260620 19:13:00.728255 26637 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bb4b8e468ccc48628004f01add882a76, State: Initialized, Role: LEARNER
I20260620 19:13:00.728291 26637 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } }
I20260620 19:13:00.728452 26637 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:00.729781 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:00.729882 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: LEARNER
I20260620 19:13:00.730014 26651 catalog_manager.cc:1259] Loaded cluster ID: e7e028618ce74e65806a6e824b3e47e0
I20260620 19:13:00.730126 26651 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: loading cluster ID for follower catalog manager: success
I20260620 19:13:00.731438 26651 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: acquiring CA information for follower catalog manager: success
I20260620 19:13:00.731734 26651 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 19:13:00.735394 26561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:00.735571 26561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:00.735603 26561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:00.736757 26561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:00.736796 26561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:00.736831 26561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:00.738382 26561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:41777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=33885
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:00.738607 26561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:00.738801 26561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.743477 26657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.743420 26659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.743417 26656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.743814 26561 server_base.cc:1034] running on GCE node
I20260620 19:13:00.744000 26561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.744205 26561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.745332 26561 hybrid_clock.cc:648] HybridClock initialized: now 1781982780745322 us; error 29 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.747666 26561 init.cc:377] Logged in from keytab as oryx/127.19.66.193@KRBTEST.COM (short username oryx)
I20260620 19:13:00.748108 26561 webserver.cc:460] Webserver started at http://127.19.66.193:33885/ using document root <none> and password file <none>
I20260620 19:13:00.748317 26561 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.748378 26561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.749673 26561 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:00.750190 26666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.750325 26561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.750403 26561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "9c1e6737a4894482a00e28e4baa20e1a"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "f7543a2e8b5f3ff38209e3153cf62f1d"
server_key_iv: "b2aa61f535d70b6ab90a4686503ed326"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.750716 26561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.772164 26561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.773844 26561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.774001 26561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.774202 26561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:00.774473 26561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:00.774505 26561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.774533 26561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:00.774555 26561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.779874 26561 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:41777
I20260620 19:13:00.779920 26779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:41777 every 8 connection(s)
I20260620 19:13:00.780242 26561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:00.785631 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26561
I20260620 19:13:00.785774 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26006
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
I20260620 19:13:00.792174 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.781262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:52187 (local address 127.19.66.253:43823)
0620 19:13:00.781302 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.781304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.781425 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:13:00.782337 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.782339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.782361 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.782389 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.783150 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.783877 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.785618 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.785934 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.785990 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.788736 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.788738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.788738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.788746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.790572 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791116 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.791182 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791456 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.791598 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.791639 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.792002 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.792085 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:13:00.792176 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.781120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:49639 (local address 127.19.66.254:46419)
0620 19:13:00.781164 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.781165 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.781278 (+ 113us) server_negotiation.cc:407] Connection header received
0620 19:13:00.782312 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.782314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.782327 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.782386 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.783112 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.783905 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.785296 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.785500 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.785556 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.789214 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.789215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.789216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.789224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.790696 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791122 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.791179 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791464 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.791595 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.791643 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.792002 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.792084 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:13:00.792588 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:33997
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:46419
--webserver_port=41457
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:00.792708 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.780924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.19.66.193:59751)
0620 19:13:00.781162 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.781201 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.782223 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.782473 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.782474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.782864 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.783023 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.783029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.784049 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.784050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.785216 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.785218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.786677 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.786678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.786740 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.787115 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.787120 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.789245 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.791460 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.791461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791634 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.791812 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.791813 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791863 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.792116 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.792117 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.792154 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.792212 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.792476 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":40,"thread_start_us":79,"threads_started":1}
I20260620 19:13:00.793051 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.780936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:59751 (local address 127.19.66.252:45677)
0620 19:13:00.781046 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.781048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.781360 (+ 312us) server_negotiation.cc:407] Connection header received
0620 19:13:00.782366 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.782367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.782379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.782430 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.783119 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.783894 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.786280 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.786560 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.786626 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.789319 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.789320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.789320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.789327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.791384 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791701 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.791756 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791922 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.791923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.791923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.792041 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.792092 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.792883 (+ 791us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.792964 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":55,"threads_started":1}
I20260620 19:13:00.793390 26785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.780925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:49639)
0620 19:13:00.781192 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.781202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.782229 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.782459 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.782463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.782817 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.783023 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.783028 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.783994 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.783995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.785216 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.785218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.785568 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.785568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.785722 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.786464 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.786473 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.789146 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.790762 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.790764 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.790764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791034 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.791243 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791243 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.791244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791389 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.791674 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.791681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.791860 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.793044 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.793300 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":117,"thread_start_us":83,"threads_started":1}
I20260620 19:13:00.793443 26780 heartbeater.cc:347] Connected to a master server at 127.19.66.252:45677
I20260620 19:13:00.793583 26780 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.793780 26780 heartbeater.cc:511] Master 127.19.66.252:45677 requested a full tablet report, sending...
I20260620 19:13:00.793778 26781 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:13:00.793869 26781 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.793942 26781 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:13:00.794024 26787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.781204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.19.66.193:52187)
0620 19:13:00.781346 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.781364 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.782240 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.782468 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.782469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.782851 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.783023 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.783029 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.783996 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.783997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.785213 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.785216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.786086 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.786087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.786139 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.786581 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.786587 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.788639 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.790660 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.790667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.790670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791028 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.791243 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.791244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.791244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.791389 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.791681 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.791682 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.791853 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.792260 (+ 407us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.793870 (+ 1610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":1560,"thread_start_us":56,"threads_started":1}
I20260620 19:13:00.794183 25810 ts_manager.cc:194] Re-registered known tserver with Master: 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:00.794306 26601 ts_manager.cc:194] Registered new tserver with Master: 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:00.794652 26782 heartbeater.cc:347] Connected to a master server at 127.19.66.253:43823
I20260620 19:13:00.794672 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:49639
I20260620 19:13:00.794695 26782 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.794747 26782 heartbeater.cc:511] Master 127.19.66.253:43823 requested a full tablet report, sending...
I20260620 19:13:00.795081 26414 ts_manager.cc:194] Registered new tserver with Master: 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
W20260620 19:13:00.873694 26788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:00.873910 26788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:00.873941 26788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:00.875070 26788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:00.875110 26788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:00.875145 26788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:00.876592 26788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:33997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=41457
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:00.876816 26788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:00.877038 26788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:00.881549 26793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.881567 26796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:00.881567 26794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:00.881778 26788 server_base.cc:1034] running on GCE node
I20260620 19:13:00.881973 26788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:00.882176 26788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:00.883308 26788 hybrid_clock.cc:648] HybridClock initialized: now 1781982780883284 us; error 37 us; skew 500 ppm
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:00.885675 26788 init.cc:377] Logged in from keytab as oryx/127.19.66.194@KRBTEST.COM (short username oryx)
I20260620 19:13:00.886072 26788 webserver.cc:460] Webserver started at http://127.19.66.194:41457/ using document root <none> and password file <none>
I20260620 19:13:00.886257 26788 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:00.886312 26788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:00.887527 26788 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:13:00.888016 26803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.888140 26788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.888216 26788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
format_stamp: "Formatted at 2026-06-20 19:12:59 on dist-test-slave-nq8r"
server_key: "2b7dd5b0f71593d0478965b87fbbc34a"
server_key_iv: "482f340a7c599d690473fdc5c81dfb16"
server_key_version: "encryptionkey@0"
I20260620 19:13:00.888521 26788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:00.910445 26788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:00.912602 26788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:00.912739 26788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:00.912993 26788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:00.913288 26788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:00.913318 26788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.913347 26788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:00.913370 26788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:00.918781 26788 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:33997
I20260620 19:13:00.918843 26916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:33997 every 8 connection(s)
I20260620 19:13:00.919154 26788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
Jun 20 19:13:00 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982780, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
I20260620 19:13:00.928289 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26788
I20260620 19:13:00.928426 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26140
I20260620 19:13:00.930780 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.919804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50907 (local address 127.19.66.254:46419)
0620 19:13:00.919850 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.919852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.920141 (+ 289us) server_negotiation.cc:407] Connection header received
0620 19:13:00.921215 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.921217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.921228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.921259 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.922928 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.923516 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924292 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.924581 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924640 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.927620 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.927622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.927622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.927631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.929056 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.929726 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.929727 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.929728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.929788 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.930056 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.930056 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.930057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.930184 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.930263 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.930614 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.930690 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:00.930960 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.919816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:50907)
0620 19:13:00.920017 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.920060 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.921095 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.921328 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.921333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.921704 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.921973 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.921979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.923614 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.923615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.924215 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.924217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.924659 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.924794 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.925496 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.925499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.927547 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.929135 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.929143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.929145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.929653 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.929854 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.929854 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.929855 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.929983 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.930292 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.930298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.930466 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.930590 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.930747 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":16,"thread_start_us":104,"threads_started":1}
I20260620 19:13:00.931418 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.920094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:44845 (local address 127.19.66.252:45677)
0620 19:13:00.920168 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.920169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.920180 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:13:00.921215 (+ 1035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.921217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.921230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.921259 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.922087 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.922823 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924340 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.924636 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924696 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.928172 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.928174 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.928174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.928181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.930199 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.930489 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.930490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.930490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.930541 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.930728 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.930729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.930730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.930844 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.930883 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.931259 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.931334 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:13:00.931432 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.919816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.19.66.194:44845)
0620 19:13:00.920051 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.920060 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.921100 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.921328 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.921333 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.921704 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.921973 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.921979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.922937 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.922938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.924217 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.924219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.924699 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.924794 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.925443 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.925454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.928059 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.930279 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.930280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.930280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.930430 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.930606 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.930607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.930608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.930662 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.931107 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.931108 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.931148 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.931230 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.931321 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":67,"spinlock_wait_cycles":8448,"thread_start_us":99,"threads_started":1}
I20260620 19:13:00.931638 26918 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:13:00.931738 26918 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.931917 26918 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:13:00.932301 25810 ts_manager.cc:194] Re-registered known tserver with Master: 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997)
I20260620 19:13:00.932744 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:50907
I20260620 19:13:00.934455 26917 heartbeater.cc:347] Connected to a master server at 127.19.66.252:45677
I20260620 19:13:00.934506 26917 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.934573 26917 heartbeater.cc:511] Master 127.19.66.252:45677 requested a full tablet report, sending...
I20260620 19:13:00.934686 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:42635
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:46419
--webserver_port=37335
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:00.934835 26601 ts_manager.cc:194] Registered new tserver with Master: 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997)
I20260620 19:13:00.934868 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.920107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.19.66.194:34183)
0620 19:13:00.920253 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:00.920262 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:00.921101 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:00.921328 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:00.921332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:00.921717 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:00.921972 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.921977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.922937 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.922938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.924217 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:00.924219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.925108 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.925109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:00.925170 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.925603 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:00.925607 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:00.927145 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:00.933969 (+ 6824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.933970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.933971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.934119 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.934282 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:00.934282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:00.934283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:00.934335 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:00.934583 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:00.934584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:00.934621 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:13:00.934682 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:13:00.934782 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":50,"thread_start_us":51,"threads_started":1}
I20260620 19:13:00.934942 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:00.920240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:34183 (local address 127.19.66.253:43823)
0620 19:13:00.920292 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:00.920294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:00.920302 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:00.921183 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:00.921184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:00.921195 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:00.921245 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:00.922054 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.922826 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.924737 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:00.924990 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:00.925053 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:00.927228 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:00.927229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:00.927229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:00.927235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:00.933900 (+ 6665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.934178 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.934179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.934179 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.934233 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:00.934381 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:00.934382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:00.934382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:00.934499 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:00.934556 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:00.934767 (+ 211us) server_negotiation.cc:299] Negotiation successful
0620 19:13:00.934857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:13:00.935945 26919 heartbeater.cc:347] Connected to a master server at 127.19.66.253:43823
I20260620 19:13:00.935994 26919 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:00.936030 26919 heartbeater.cc:511] Master 127.19.66.253:43823 requested a full tablet report, sending...
I20260620 19:13:00.936290 26414 ts_manager.cc:194] Registered new tserver with Master: 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997)
W20260620 19:13:01.018054 26924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:01.018241 26924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:01.018271 26924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:01.019425 26924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:01.019469 26924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:01.019503 26924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:01.020961 26924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:42635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=37335
--tserver_master_addrs=127.19.66.254:46419,127.19.66.253:43823,127.19.66.252:45677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:01.021196 26924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:01.021394 26924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:01.025874 26932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:01.025986 26924 server_base.cc:1034] running on GCE node
W20260620 19:13:01.026021 26930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:01.026062 26929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:01.026309 26924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:01.026517 26924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:01.027650 26924 hybrid_clock.cc:648] HybridClock initialized: now 1781982781027636 us; error 25 us; skew 500 ppm
Jun 20 19:13:01 dist-test-slave-nq8r krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982781, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:01.030055 26924 init.cc:377] Logged in from keytab as oryx/127.19.66.195@KRBTEST.COM (short username oryx)
I20260620 19:13:01.030457 26924 webserver.cc:460] Webserver started at http://127.19.66.195:37335/ using document root <none> and password file <none>
I20260620 19:13:01.030637 26924 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:01.030684 26924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:01.032022 26924 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:01.032546 26939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:01.032682 26924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:01.032761 26924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "3aa0eeb2e8dc409b98093512e408b54c"
format_stamp: "Formatted at 2026-06-20 19:13:00 on dist-test-slave-nq8r"
server_key: "f2a7d6d931cf5af6b43ac46953c464fd"
server_key_iv: "1411b536a5ef7c0fbe4dc9307c69c781"
server_key_version: "encryptionkey@0"
I20260620 19:13:01.033123 26924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:01.052743 26924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:01.054544 26924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:01.054666 26924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:01.054870 26924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:01.055140 26924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:01.055171 26924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:01.055190 26924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:01.055213 26924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:01.060508 26924 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:42635
I20260620 19:13:01.060554 27052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:42635 every 8 connection(s)
I20260620 19:13:01.060913 26924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:13:01 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982781, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
Jun 20 19:13:01 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982781, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.252@KRBTEST.COM
Jun 20 19:13:01 dist-test-slave-nq8r krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982781, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.253@KRBTEST.COM
I20260620 19:13:01.069170 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 26924
I20260620 19:13:01.072288 27057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.061615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:48443)
0620 19:13:01.061897 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.061907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.062939 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.063177 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.063180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.063574 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:01.063755 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.063760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.064453 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.064454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.065550 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.065552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.066023 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.066161 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.066721 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:01.066731 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:01.068243 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:01.069918 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.069924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.069927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.070194 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.070418 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.070419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.070420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.070555 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.070854 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:01.070861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:01.071048 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.071155 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.071331 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":27,"thread_start_us":86,"threads_started":1}
I20260620 19:13:01.072372 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.062047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:38967 (local address 127.19.66.252:45677)
0620 19:13:01.062117 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.062119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.062126 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:01.063038 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.063040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.063052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.063082 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.064445 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.064965 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.065822 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.066054 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066118 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.068843 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.068867 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.068868 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.068876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.071345 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.071700 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.071702 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.071702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.071751 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.071914 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.071915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.071915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.072011 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.072065 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.072204 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.072276 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 19:13:01.072405 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.061896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.19.66.195:38967)
0620 19:13:01.062065 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.062073 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.062938 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.063177 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.063180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.063565 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:01.063755 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.063760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.065043 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.065044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.065755 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.065757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.066116 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.066187 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.066767 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:01.066772 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:01.068735 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:01.071420 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.071421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.071422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.071636 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.071803 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.071804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.071804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.071859 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.072064 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:01.072064 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:01.072104 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.072176 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.072276 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":3,"thread_start_us":53,"threads_started":1}
I20260620 19:13:01.072996 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.061611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:48443 (local address 127.19.66.254:46419)
0620 19:13:01.061659 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.061661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.061974 (+ 313us) server_negotiation.cc:407] Connection header received
0620 19:13:01.063038 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.063040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.063052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.063082 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.063851 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.064358 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.065643 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.065943 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066014 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.068333 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.068335 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.068336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.068345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.069849 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.070275 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.070277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.070277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.070340 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.070617 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.070618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.070618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.070762 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.070827 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.072788 (+ 1961us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.072894 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:13:01.073167 27053 heartbeater.cc:347] Connected to a master server at 127.19.66.252:45677
I20260620 19:13:01.073259 27053 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:01.073447 27053 heartbeater.cc:511] Master 127.19.66.252:45677 requested a full tablet report, sending...
I20260620 19:13:01.073725 27054 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46419
I20260620 19:13:01.073768 27054 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:01.073819 27054 heartbeater.cc:511] Master 127.19.66.254:46419 requested a full tablet report, sending...
I20260620 19:13:01.073887 26601 ts_manager.cc:194] Registered new tserver with Master: 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195:42635)
I20260620 19:13:01.074033 25810 ts_manager.cc:194] Re-registered known tserver with Master: 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195:42635)
I20260620 19:13:01.074292 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.061647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:56987 (local address 127.19.66.253:43823)
0620 19:13:01.061685 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.061687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.061894 (+ 207us) server_negotiation.cc:407] Connection header received
0620 19:13:01.063029 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.063031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.063044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.063090 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.064931 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.065432 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066342 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.066587 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066646 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.069183 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.069185 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.069185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.069193 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.072061 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.073539 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.073540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.073540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.073597 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.073785 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.073786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.073786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.073902 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.073952 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.074130 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.074206 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:01.074440 25810 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:48443
I20260620 19:13:01.074951 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.061615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.19.66.195:56987)
0620 19:13:01.061769 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.061805 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.062920 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.063176 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.063180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.063549 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:01.063753 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.063757 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.065508 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.065509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.066260 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.066261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.066646 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.066647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.066728 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.067091 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:01.067095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:01.069084 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:01.073259 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.073260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.073261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.073452 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.073667 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:01.073667 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:01.073668 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:01.073722 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:01.073993 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:01.073993 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:01.074030 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.074482 (+ 452us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.074578 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260620 19:13:01.075264 27055 heartbeater.cc:347] Connected to a master server at 127.19.66.253:43823
I20260620 19:13:01.075313 27055 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:01.075376 27055 heartbeater.cc:511] Master 127.19.66.253:43823 requested a full tablet report, sending...
I20260620 19:13:01.075635 26414 ts_manager.cc:194] Registered new tserver with Master: 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195:42635)
I20260620 19:13:01.081212 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.070747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.19.66.252:45677)
0620 19:13:01.071190 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.071191 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.071197 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:13:01.071215 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.071216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.071226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.071274 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.072054 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.074059 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.075025 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.075304 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.075363 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.078483 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.078484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.078485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.078491 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.080000 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.080347 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.080348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.080349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.080406 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.080595 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.080596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.080596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.080742 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.080838 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.081039 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.081118 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":51,"threads_started":1}
I20260620 19:13:01.081590 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.071481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.19.66.254:46419)
0620 19:13:01.073061 (+ 1580us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.073062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.073068 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:13:01.073085 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.073086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.073094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.073135 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.075449 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.076165 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.077152 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.077327 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.077388 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.079176 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.079178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.079178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.079185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.080474 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.080832 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.080833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.080833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.080910 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.081110 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.081110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.081111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.081219 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.081275 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.081462 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.081531 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1568,"thread_start_us":85,"threads_started":1}
I20260620 19:13:01.081903 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.070689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49372 (local address 127.19.66.253:43823)
0620 19:13:01.074373 (+ 3684us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.074375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.074381 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:13:01.074398 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.074399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.074408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.074458 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.075739 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.076434 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.077654 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.077860 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.077927 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.079354 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.079356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.079356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.079363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.080730 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.081059 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.081060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.081060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.081113 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.081378 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.081378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.081379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.081507 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.081571 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.081745 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.081816 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3671,"thread_start_us":53,"threads_started":1}
I20260620 19:13:01.083092 25810 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:01.084749 25810 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:01.091650 27074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.087800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41777 (local address 127.0.0.1:60532)
0620 19:13:01.087910 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.087917 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.087953 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.088536 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.088537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.088539 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.088571 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.088572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089930 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.089931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.090688 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.090690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091422 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.091424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.091492 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.091496 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.091534 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.091574 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260620 19:13:01.091665 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.088221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.19.66.193:41777)
0620 19:13:01.088399 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.088403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.088411 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:01.088446 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.088447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.088462 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.088531 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.088625 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089865 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.090758 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091318 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.091411 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.091525 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.091560 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.091594 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260620 19:13:01.091745 26368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.087766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33997 (local address 127.0.0.1:56976)
0620 19:13:01.087808 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.087816 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.087849 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.088351 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.088352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.088354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.088396 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.088433 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089720 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.089721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.090706 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.090707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091528 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.091529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.091589 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.091592 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.091621 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.091665 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":7}
I20260620 19:13:01.092064 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.088041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.19.66.194:33997)
0620 19:13:01.088185 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.088189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.088199 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:01.088236 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.088238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.088253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.088509 (+ 256us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.088564 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089640 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.090780 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091436 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.091837 (+ 401us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.091918 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.091944 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.091984 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 19:13:01.092300 26714 tablet_service.cc:1467] Processing CreateTablet for tablet ebf38f08c5b047a2914a84018c258e48 (DEFAULT_TABLE table=test-table [id=170a1b7bd558432ca8d0c3a654809e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:01.092427 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.087809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:42635 (local address 127.0.0.1:41786)
0620 19:13:01.088019 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.088028 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.088075 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.088318 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.088319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.088321 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.088366 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.088368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089353 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.089354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.090609 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.090611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091539 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.091540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.091589 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.091595 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.091621 (+ 26us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.091667 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260620 19:13:01.092640 26714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf38f08c5b047a2914a84018c258e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:01.092722 26851 tablet_service.cc:1467] Processing CreateTablet for tablet ebf38f08c5b047a2914a84018c258e48 (DEFAULT_TABLE table=test-table [id=170a1b7bd558432ca8d0c3a654809e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:01.092957 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.088006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41786 (local address 127.19.66.195:42635)
0620 19:13:01.088147 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.088152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.088161 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:01.088198 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.088200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.088216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.088434 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.088489 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.089284 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.090686 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.091433 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.092695 (+ 1262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.092780 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.092805 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.092881 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260620 19:13:01.093029 26851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf38f08c5b047a2914a84018c258e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:01.093415 26987 tablet_service.cc:1467] Processing CreateTablet for tablet ebf38f08c5b047a2914a84018c258e48 (DEFAULT_TABLE table=test-table [id=170a1b7bd558432ca8d0c3a654809e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:01.093676 26987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf38f08c5b047a2914a84018c258e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:01.096100 27079 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Bootstrap starting.
I20260620 19:13:01.096371 27081 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Bootstrap starting.
I20260620 19:13:01.096711 27079 tablet_bootstrap.cc:654] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:01.096947 27081 tablet_bootstrap.cc:654] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:01.096997 27080 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Bootstrap starting.
I20260620 19:13:01.097029 27079 log.cc:824] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:01.097282 27081 log.cc:824] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:01.097553 27080 tablet_bootstrap.cc:654] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:01.097828 27080 log.cc:824] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:01.098095 27079 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: No bootstrap required, opened a new log
I20260620 19:13:01.098114 27081 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: No bootstrap required, opened a new log
I20260620 19:13:01.098168 27081 ts_tablet_manager.cc:1397] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:01.098165 27079 ts_tablet_manager.cc:1397] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:13:01.098495 27080 tablet_bootstrap.cc:492] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: No bootstrap required, opened a new log
I20260620 19:13:01.098556 27080 ts_tablet_manager.cc:1397] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:01.099581 27079 raft_consensus.cc:348] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.099700 27079 raft_consensus.cc:374] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:01.099720 27079 raft_consensus.cc:729] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa0eeb2e8dc409b98093512e408b54c, State: Initialized, Role: FOLLOWER
I20260620 19:13:01.099804 27079 consensus_queue.cc:260] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.100013 27079 ts_tablet_manager.cc:1428] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:13:01.100080 27054 heartbeater.cc:503] Master 127.19.66.254:46419 was elected leader, sending a full tablet report...
I20260620 19:13:01.100032 27081 raft_consensus.cc:348] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.100178 27081 raft_consensus.cc:374] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:01.100206 27081 raft_consensus.cc:729] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cc1ae5b92540dc91fb5ce15fe15f4e, State: Initialized, Role: FOLLOWER
I20260620 19:13:01.100311 27081 consensus_queue.cc:260] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.100358 27080 raft_consensus.cc:348] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.100483 27080 raft_consensus.cc:374] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:01.100524 27080 raft_consensus.cc:729] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1e6737a4894482a00e28e4baa20e1a, State: Initialized, Role: FOLLOWER
I20260620 19:13:01.100531 27081 ts_tablet_manager.cc:1428] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:13:01.100623 27080 consensus_queue.cc:260] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.100661 26918 heartbeater.cc:503] Master 127.19.66.254:46419 was elected leader, sending a full tablet report...
I20260620 19:13:01.101293 26781 heartbeater.cc:503] Master 127.19.66.254:46419 was elected leader, sending a full tablet report...
I20260620 19:13:01.102152 27080 ts_tablet_manager.cc:1428] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 19:13:01.140471 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.138037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.19.66.252:45677)
0620 19:13:01.138094 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.138096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.138104 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:01.138134 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.138134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.138156 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.138212 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.138354 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.138857 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.139725 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.140102 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.140152 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.140206 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.140368 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.140414 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:13:01.140484 27075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.137957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47406)
0620 19:13:01.138021 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.138032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.138079 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.138218 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.138219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.138221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.138275 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.138276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.138931 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.138931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.139659 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.139661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.140239 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.140240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.140292 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.140296 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.140379 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.140420 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:13:01.140821 26611 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 19:13:01.141649 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f50f03620cc74558967c7da487a3f08d. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:01.141748 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: LEARNER
W20260620 19:13:01.170028 26920 tablet.cc:2369] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:01.178438 27085 raft_consensus.cc:482] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:01.178556 27085 raft_consensus.cc:504] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.178905 27085 leader_election.cc:288] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997), 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:01.181540 27078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.179103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:46779 (local address 127.19.66.193:41777)
0620 19:13:01.179149 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.179150 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.179232 (+ 82us) server_negotiation.cc:407] Connection header received
0620 19:13:01.179275 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.179276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.179286 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.179339 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.179463 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.179932 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.180794 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.181219 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.181283 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.181294 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.181438 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.181476 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:13:01.181555 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.179141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:37293 (local address 127.19.66.194:33997)
0620 19:13:01.179181 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.179183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.179191 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:01.179224 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.179224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.179233 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.179281 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:01.179424 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.179898 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.180785 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.181278 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.181342 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.181350 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.181468 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.181501 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:13:01.181555 27059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.179051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41777 (local address 127.19.66.195:46779)
0620 19:13:01.179170 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.179180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.179229 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.179344 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.179344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.179345 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.179401 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.179403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.180000 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.180001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.180712 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.180714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.181282 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.181283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.181357 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.181360 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.181414 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.181452 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":112}
I20260620 19:13:01.181702 27058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.179050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33997 (local address 127.19.66.195:37293)
0620 19:13:01.179108 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:01.179119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:01.179178 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:01.179290 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:01.179291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:01.179292 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:01.179365 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.179367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.179968 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.179969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.180709 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:01.180711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.181341 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.181342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:01.181409 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.181413 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:01.181462 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:13:01.181498 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:13:01.181918 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf38f08c5b047a2914a84018c258e48" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1e6737a4894482a00e28e4baa20e1a" is_pre_election: true
I20260620 19:13:01.181959 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf38f08c5b047a2914a84018c258e48" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" is_pre_election: true
I20260620 19:13:01.182073 26734 raft_consensus.cc:2454] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 0.
I20260620 19:13:01.182073 26871 raft_consensus.cc:2454] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 0.
I20260620 19:13:01.182276 26942 leader_election.cc:302] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12cc1ae5b92540dc91fb5ce15fe15f4e, 3aa0eeb2e8dc409b98093512e408b54c; no voters:
I20260620 19:13:01.182428 27085 raft_consensus.cc:2781] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:01.182475 27085 raft_consensus.cc:482] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:01.182499 27085 raft_consensus.cc:3037] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:01.183183 27085 raft_consensus.cc:504] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.183295 27085 leader_election.cc:288] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 election: Requested vote from peers 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997), 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:01.183466 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf38f08c5b047a2914a84018c258e48" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
I20260620 19:13:01.183481 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf38f08c5b047a2914a84018c258e48" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1e6737a4894482a00e28e4baa20e1a"
I20260620 19:13:01.183537 26871 raft_consensus.cc:3037] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:01.183547 26734 raft_consensus.cc:3037] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:01.184185 26734 raft_consensus.cc:2454] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 1.
I20260620 19:13:01.184288 26871 raft_consensus.cc:2454] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 1.
I20260620 19:13:01.184340 26941 leader_election.cc:302] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa0eeb2e8dc409b98093512e408b54c, 9c1e6737a4894482a00e28e4baa20e1a; no voters:
I20260620 19:13:01.184432 27085 raft_consensus.cc:2781] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:01.184572 27085 raft_consensus.cc:686] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 1 LEADER]: Becoming Leader. State: Replica: 3aa0eeb2e8dc409b98093512e408b54c, State: Running, Role: LEADER
I20260620 19:13:01.184674 27085 consensus_queue.cc:237] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } }
I20260620 19:13:01.185669 25810 catalog_manager.cc:5495] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195). New cstate: current_term: 1 leader_uuid: "3aa0eeb2e8dc409b98093512e408b54c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:01.186231 27093 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: NON_VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: true } } }
I20260620 19:13:01.186323 27093 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:01.186693 26565 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: attempting to promote NON_VOTER bb4b8e468ccc48628004f01add882a76 to VOTER
I20260620 19:13:01.186879 26565 consensus_queue.cc:237] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } }
I20260620 19:13:01.187294 26610 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 LEARNER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 19:13:01.187557 26424 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Refusing update from remote peer f50f03620cc74558967c7da487a3f08d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 19:13:01.187693 26369 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260620 19:13:01.187800 27095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [LEADER]: Connected to new peer: Peer: permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 19:13:01.188344 26565 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bb4b8e468ccc48628004f01add882a76 (127.19.66.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.188472 26424 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bb4b8e468ccc48628004f01add882a76 (127.19.66.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.188488 26611 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, bb4b8e468ccc48628004f01add882a76 (127.19.66.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189236 26565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189304 26565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:01.189258 26640 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189330 26640 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:13:01.189360 26565 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189404 26565 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:01.189404 27095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189527 27095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:01.189502 27096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f50f03620cc74558967c7da487a3f08d" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:01.189568 27096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:13:01.208570 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.206519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.19.66.195:42635)
0620 19:13:01.206576 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.206578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.206586 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:01.206619 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.206620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.206637 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.206696 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:01.206855 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.207319 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.208005 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.208178 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.208222 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.208321 (+ 99us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:01.208419 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:01.208476 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.208488 (+ 12us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.208505 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:13:01.210328 26871 raft_consensus.cc:1261] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Refusing update from remote peer 3aa0eeb2e8dc409b98093512e408b54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:01.210335 26734 raft_consensus.cc:1261] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Refusing update from remote peer 3aa0eeb2e8dc409b98093512e408b54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:01.210654 27085 consensus_queue.cc:1035] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:01.210800 27091 consensus_queue.cc:1035] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:01.213457 27100 mvcc.cc:205] Tried to move back new op lower bound from 7299001471835115520 to 7299001471733022720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001471835115520}]
I20260620 19:13:01.213825 27099 mvcc.cc:205] Tried to move back new op lower bound from 7299001471835115520 to 7299001471733022720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001471835115520}]
I20260620 19:13:01.214756 27098 mvcc.cc:205] Tried to move back new op lower bound from 7299001471835115520 to 7299001471733022720. Current Snapshot: MvccSnapshot[applied={T|T < 7299001471835115520 or (T in {7299001471835115520})}]
I20260620 19:13:01.215744 25810 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:01.215837 25810 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:01.217044 25810 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f50f03620cc74558967c7da487a3f08d: Sending DeleteTablet for 3 replicas of tablet ebf38f08c5b047a2914a84018c258e48
I20260620 19:13:01.217344 26851 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf38f08c5b047a2914a84018c258e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:01 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:56976
I20260620 19:13:01.217428 26714 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf38f08c5b047a2914a84018c258e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:01 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:60532
I20260620 19:13:01.217676 27108 tablet_replica.cc:330] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: stopping tablet replica
I20260620 19:13:01.217775 27108 raft_consensus.cc:2229] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:01.217869 27108 raft_consensus.cc:2258] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:01.218165 26987 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf38f08c5b047a2914a84018c258e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:01 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:41786
I20260620 19:13:01.218319 27108 ts_tablet_manager.cc:1905] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:01.218437 27110 tablet_replica.cc:330] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: stopping tablet replica
I20260620 19:13:01.218673 27110 raft_consensus.cc:2229] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:01.218905 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 25778
I20260620 19:13:01.219925 27108 ts_tablet_manager.cc:1918] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:01.220001 27108 log.cc:1196] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/ebf38f08c5b047a2914a84018c258e48
I20260620 19:13:01.220258 27108 ts_tablet_manager.cc:1939] T ebf38f08c5b047a2914a84018c258e48 P 9c1e6737a4894482a00e28e4baa20e1a: Deleting consensus metadata
I20260620 19:13:01.220220 27110 raft_consensus.cc:2258] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:01.221468 27110 ts_tablet_manager.cc:1905] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:01.223227 27110 ts_tablet_manager.cc:1918] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:01.223344 27110 log.cc:1196] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/ebf38f08c5b047a2914a84018c258e48
I20260620 19:13:01.223630 27110 ts_tablet_manager.cc:1939] T ebf38f08c5b047a2914a84018c258e48 P 3aa0eeb2e8dc409b98093512e408b54c: Deleting consensus metadata
I20260620 19:13:01.224107 27107 tablet_replica.cc:330] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: stopping tablet replica
I20260620 19:13:01.224255 27107 raft_consensus.cc:2229] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:01.224406 27107 raft_consensus.cc:2258] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:01.224830 27107 ts_tablet_manager.cc:1905] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:13:01.225334 26940 connection.cc:535] server connection from 127.0.0.1:41786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 19:13:01.225399 26669 connection.cc:535] server connection from 127.0.0.1:60532 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:13:01.226791 27107 ts_tablet_manager.cc:1918] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:01.226862 27107 log.cc:1196] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/ebf38f08c5b047a2914a84018c258e48
I20260620 19:13:01.227124 27107 ts_tablet_manager.cc:1939] T ebf38f08c5b047a2914a84018c258e48 P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting consensus metadata
W20260620 19:13:01.227451 26806 connection.cc:441] server connection from 127.0.0.1:56976 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56976 (request call id 1) could send its response
I20260620 19:13:01.228181 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.218372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.19.66.252:45677)
0620 19:13:01.218411 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.218413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.218602 (+ 189us) server_negotiation.cc:407] Connection header received
0620 19:13:01.218851 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.218852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.218862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.218889 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.221146 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.221832 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.222809 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.223094 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.223138 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.224900 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.224901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.224902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.224909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.226866 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.227265 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.227266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.227267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.227322 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.227663 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.227664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.227665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.227783 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.227855 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.228027 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.228096 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:01.235291 26453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.229048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49386 (local address 127.19.66.253:43823)
0620 19:13:01.229088 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.229090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.229182 (+ 92us) server_negotiation.cc:407] Connection header received
0620 19:13:01.229205 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.229207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.229216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.229283 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.229866 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.230347 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.231106 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.231301 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.231354 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.232675 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.232676 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.232677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.232684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.234173 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.234536 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.234537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.234538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.234597 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.234781 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.234782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.234782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.234912 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.235007 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.235117 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.235193 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:01.236003 27067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:01.229132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.19.66.252:45677)
0620 19:13:01.229166 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:01.229167 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:01.229290 (+ 123us) server_negotiation.cc:407] Connection header received
0620 19:13:01.229356 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:01.229357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:01.229366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:01.229414 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:01.230023 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.230766 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.231806 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:01.232082 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:01.232152 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:01.233434 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:01.233436 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:01.233436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:01.233443 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:01.234860 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.235186 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.235187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.235188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.235241 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:01.235433 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:01.235434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:01.235434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:01.235561 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:01.235626 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:01.235824 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 19:13:01.235902 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:13:02.214581 27117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.214297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:41707)
0620 19:13:02.214418 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.214503 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260620 19:13:02.214687 27118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.214445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:46521)
0620 19:13:02.214557 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.214634 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
W20260620 19:13:02.215013 26781 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:46419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
W20260620 19:13:02.215207 26918 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:46419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.215560 27119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.215364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:47609)
0620 19:13:02.215463 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.215522 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":38,"threads_started":1}
W20260620 19:13:02.215723 27054 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:46419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.718206 27120 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f50f03620cc74558967c7da487a3f08d)
I20260620 19:13:02.718221 27121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f50f03620cc74558967c7da487a3f08d)
I20260620 19:13:02.718321 27121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } }
I20260620 19:13:02.718367 27120 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } }
I20260620 19:13:02.718750 27121 leader_election.cc:288] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419), bb4b8e468ccc48628004f01add882a76 (127.19.66.252:45677)
I20260620 19:13:02.718972 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47422)
0620 19:13:02.718856 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.718912 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
W20260620 19:13:02.719199 26588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.718986 27120 leader_election.cc:288] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419), 10deb988d4764b8997e2005e3543510f (127.19.66.253:43823)
I20260620 19:13:02.719230 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47434)
0620 19:13:02.719142 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.719192 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":31,"threads_started":1}
W20260620 19:13:02.719328 26401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.719710 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.719633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47440)
0620 19:13:02.719666 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.719687 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:13:02.719715 27123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.719632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47456)
0620 19:13:02.719669 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.719691 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:13:02.719772 26588 leader_election.cc:334] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419): Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
W20260620 19:13:02.719770 26401 leader_election.cc:334] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419): Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.721742 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45677 (local address 127.0.0.1:47434)
0620 19:13:02.719121 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.719131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:02.719193 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:02.719342 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:02.719342 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:02.719345 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:02.719400 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.719402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.720198 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.720199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.721023 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.721025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.721525 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.721586 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.721589 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:02.721642 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:13:02.721683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":42,"threads_started":1}
I20260620 19:13:02.721740 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47434 (local address 127.19.66.252:45677)
0620 19:13:02.719103 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.719105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.719180 (+ 75us) server_negotiation.cc:407] Connection header received
0620 19:13:02.719243 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.719244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.719253 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.719328 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:02.719473 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.720132 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721100 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.721462 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721518 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.721529 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.721646 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.721679 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":48,"threads_started":1}
I20260620 19:13:02.721993 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10deb988d4764b8997e2005e3543510f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb4b8e468ccc48628004f01add882a76" is_pre_election: true
I20260620 19:13:02.722105 26611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 10deb988d4764b8997e2005e3543510f in term 1.
I20260620 19:13:02.722123 27126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49398 (local address 127.19.66.253:43823)
0620 19:13:02.719046 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.719047 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.719056 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:02.719597 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.719598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.719608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.719652 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:02.719988 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.720449 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721402 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.721809 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721877 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.721887 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.722040 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.722073 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":47,"threads_started":1}
I20260620 19:13:02.722172 27124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.718774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:43823 (local address 127.0.0.1:49398)
0620 19:13:02.718991 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.719000 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:02.719059 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:02.719878 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:02.719878 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:02.719880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:02.719917 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.719919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.720520 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.720521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.721328 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.721331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.721919 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.721920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.721975 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.721978 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:02.722024 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:13:02.722072 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":63,"threads_started":1}
I20260620 19:13:02.722266 26398 leader_election.cc:302] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10deb988d4764b8997e2005e3543510f, bb4b8e468ccc48628004f01add882a76; no voters: f50f03620cc74558967c7da487a3f08d
I20260620 19:13:02.722368 27121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 19:13:02.722332 26424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bb4b8e468ccc48628004f01add882a76" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "10deb988d4764b8997e2005e3543510f" is_pre_election: true
I20260620 19:13:02.722409 27121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Starting leader election (detected failure of leader f50f03620cc74558967c7da487a3f08d)
I20260620 19:13:02.722429 27121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:13:02.723115 27121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } }
I20260620 19:13:02.723227 27121 leader_election.cc:288] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 election: Requested vote from peers f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419), bb4b8e468ccc48628004f01add882a76 (127.19.66.252:45677)
I20260620 19:13:02.723235 26424 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bb4b8e468ccc48628004f01add882a76 in current term 2: Already voted for candidate 10deb988d4764b8997e2005e3543510f in this term.
I20260620 19:13:02.723438 26585 leader_election.cc:302] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bb4b8e468ccc48628004f01add882a76; no voters: 10deb988d4764b8997e2005e3543510f, f50f03620cc74558967c7da487a3f08d
I20260620 19:13:02.723492 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.723352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47460)
0620 19:13:02.723447 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.723471 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 19:13:02.723479 26611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "10deb988d4764b8997e2005e3543510f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bb4b8e468ccc48628004f01add882a76"
I20260620 19:13:02.723541 26611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:13:02.723768 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.723691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47464)
0620 19:13:02.723731 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.723753 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 19:13:02.723819 26401 leader_election.cc:334] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419): Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
I20260620 19:13:02.724212 26611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10deb988d4764b8997e2005e3543510f in term 2.
I20260620 19:13:02.724288 27120 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:13:02.724360 26398 leader_election.cc:302] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 10deb988d4764b8997e2005e3543510f, bb4b8e468ccc48628004f01add882a76; no voters: f50f03620cc74558967c7da487a3f08d
I20260620 19:13:02.724424 27121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:13:02.724550 27121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [term 2 LEADER]: Becoming Leader. State: Replica: 10deb988d4764b8997e2005e3543510f, State: Running, Role: LEADER
I20260620 19:13:02.724615 27121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } }
I20260620 19:13:02.724917 27130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10deb988d4764b8997e2005e3543510f. Latest consensus state: current_term: 2 leader_uuid: "10deb988d4764b8997e2005e3543510f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:02.725003 27130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:02.725174 27132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:02.725556 27132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:02.725613 27132 catalog_manager.cc:1259] Loaded cluster ID: e7e028618ce74e65806a6e824b3e47e0
I20260620 19:13:02.725641 27132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:02.725791 27132 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:02.725857 27132 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Loaded TSK: 0
I20260620 19:13:02.725991 27132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:13:02.768357 26414 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49386:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:02.769938 26414 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:02.770829 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.770734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47494)
0620 19:13:02.770774 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.770803 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:13:02.770844 26611 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [term 2 FOLLOWER]: Refusing update from remote peer 10deb988d4764b8997e2005e3543510f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 19:13:02.771082 27121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 19:13:02.771209 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.771126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.0.0.1:47510)
0620 19:13:02.771163 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.771186 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:13:02.771272 26401 consensus_peers.cc:487] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f -> Peer f50f03620cc74558967c7da487a3f08d (127.19.66.254:46419): Couldn't send request to peer f50f03620cc74558967c7da487a3f08d. Status: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 19:13:02.771625 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10deb988d4764b8997e2005e3543510f. Latest consensus state: current_term: 2 leader_uuid: "10deb988d4764b8997e2005e3543510f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:02.771711 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:13:02.772099 27121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "10deb988d4764b8997e2005e3543510f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:02.772181 27121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:02.772253 27120 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "10deb988d4764b8997e2005e3543510f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f50f03620cc74558967c7da487a3f08d" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46419 } } peers { permanent_uuid: "10deb988d4764b8997e2005e3543510f" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 43823 } attrs { promote: false } } peers { permanent_uuid: "bb4b8e468ccc48628004f01add882a76" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45677 } attrs { promote: false } } }
I20260620 19:13:02.772405 27120 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4b8e468ccc48628004f01add882a76 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:13:02.776830 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.773842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33997 (local address 127.0.0.1:56980)
0620 19:13:02.774003 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.774010 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:02.774032 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:02.774218 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:02.774219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:02.774221 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:02.774280 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.774281 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.774992 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.774993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.775989 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.775991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.776612 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.776613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.776670 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.776675 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:02.776729 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:13:02.776765 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260620 19:13:02.776868 27140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.774018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56980 (local address 127.19.66.194:33997)
0620 19:13:02.774122 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.774124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.774131 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:02.774155 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.774156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.774165 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.774217 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:02.774349 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.774926 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.776069 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.776536 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.776605 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.776617 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.776739 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.776778 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":64,"threads_started":1}
I20260620 19:13:02.777098 26851 tablet_service.cc:1467] Processing CreateTablet for tablet a13a39aebaed4215bb0bc38cd4364dca (DEFAULT_TABLE table=test-table [id=9ea6b667848e415eba9dc550ad4c58a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:02.777231 26851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a39aebaed4215bb0bc38cd4364dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:02.777233 27122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.773819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:42635 (local address 127.0.0.1:41796)
0620 19:13:02.773894 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.773904 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:02.773950 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:02.774259 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:02.774260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:02.774263 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:02.774307 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.774308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.775222 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.775222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.776273 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.776275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.777019 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.777020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.777084 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.777087 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:02.777134 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:13:02.777171 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":25}
I20260620 19:13:02.777248 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.773907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.19.66.195:42635)
0620 19:13:02.774017 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.774019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.774028 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:02.774054 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.774055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.774066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.774151 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:02.774385 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.775137 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.776356 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.776913 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.777016 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.777028 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.777144 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.777185 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260620 19:13:02.777508 26987 tablet_service.cc:1467] Processing CreateTablet for tablet a13a39aebaed4215bb0bc38cd4364dca (DEFAULT_TABLE table=test-table [id=9ea6b667848e415eba9dc550ad4c58a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:02.777617 26987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a39aebaed4215bb0bc38cd4364dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:02.778436 27142 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Bootstrap starting.
I20260620 19:13:02.778694 27138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.773885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41777 (local address 127.0.0.1:60536)
0620 19:13:02.774019 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.774029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:02.774075 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:02.774407 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:02.774408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:02.774409 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:02.774436 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.774438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.775295 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.775296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.777787 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:02.777789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.778504 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.778505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:02.778562 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.778567 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:02.778594 (+ 27us) client_negotiation.cc:240] Negotiation successful
0620 19:13:02.778632 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 19:13:02.778918 27143 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Bootstrap starting.
I20260620 19:13:02.778947 27141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.774177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.19.66.193:41777)
0620 19:13:02.774273 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.774275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.774281 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:13:02.774304 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.774304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.774314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.774368 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:02.774528 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.775225 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.777871 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.778394 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.778798 (+ 404us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.778812 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.778840 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.778880 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":54,"threads_started":1}
I20260620 19:13:02.779150 27142 tablet_bootstrap.cc:654] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:02.779141 26714 tablet_service.cc:1467] Processing CreateTablet for tablet a13a39aebaed4215bb0bc38cd4364dca (DEFAULT_TABLE table=test-table [id=9ea6b667848e415eba9dc550ad4c58a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:02.779247 26714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a13a39aebaed4215bb0bc38cd4364dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:02.779279 27143 tablet_bootstrap.cc:654] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:02.779728 27142 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: No bootstrap required, opened a new log
I20260620 19:13:02.779771 27142 ts_tablet_manager.cc:1397] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:02.779870 27143 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: No bootstrap required, opened a new log
I20260620 19:13:02.779894 27142 raft_consensus.cc:348] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.779959 27142 raft_consensus.cc:374] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:02.779978 27142 raft_consensus.cc:729] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cc1ae5b92540dc91fb5ce15fe15f4e, State: Initialized, Role: FOLLOWER
I20260620 19:13:02.779938 27143 ts_tablet_manager.cc:1397] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:02.780023 27142 consensus_queue.cc:260] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.780177 27142 ts_tablet_manager.cc:1428] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:02.780155 27143 raft_consensus.cc:348] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.780630 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.780446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:46613)
0620 19:13:02.780568 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.780600 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260620 19:13:02.780743 27148 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a: Bootstrap starting.
I20260620 19:13:02.781000 26919 heartbeater.cc:503] Master 127.19.66.253:43823 was elected leader, sending a full tablet report...
I20260620 19:13:02.780993 27143 raft_consensus.cc:374] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:02.781068 27143 raft_consensus.cc:729] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa0eeb2e8dc409b98093512e408b54c, State: Initialized, Role: FOLLOWER
I20260620 19:13:02.781141 27143 consensus_queue.cc:260] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.781278 27148 tablet_bootstrap.cc:654] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:02.781350 27143 ts_tablet_manager.cc:1428] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:13:02.781561 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.780821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:53537)
0620 19:13:02.781507 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.781537 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669}
I20260620 19:13:02.781783 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.781554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:48317)
0620 19:13:02.781717 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.781751 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260620 19:13:02.781891 27148 tablet_bootstrap.cc:492] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a: No bootstrap required, opened a new log
I20260620 19:13:02.781935 27148 ts_tablet_manager.cc:1397] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:02.782102 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.782016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:53699)
0620 19:13:02.782059 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.782081 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:13:02.782073 27148 raft_consensus.cc:348] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.782135 27148 raft_consensus.cc:374] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:02.782152 27148 raft_consensus.cc:729] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1e6737a4894482a00e28e4baa20e1a, State: Initialized, Role: FOLLOWER
I20260620 19:13:02.782188 27148 consensus_queue.cc:260] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.782228 27055 heartbeater.cc:503] Master 127.19.66.253:43823 was elected leader, sending a full tablet report...
I20260620 19:13:02.782328 27148 ts_tablet_manager.cc:1428] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:02.782763 26782 heartbeater.cc:503] Master 127.19.66.253:43823 was elected leader, sending a full tablet report...
I20260620 19:13:02.782816 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.782574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:49583)
0620 19:13:02.782748 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.782782 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":74,"threads_started":1}
I20260620 19:13:02.783095 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.783002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:37647)
0620 19:13:02.783048 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.783072 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:13:02.812388 27056 tablet.cc:2369] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:02.869580 27149 raft_consensus.cc:482] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:02.869683 27149 raft_consensus.cc:504] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.869817 27149 leader_election.cc:288] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997), 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:02.870030 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a39aebaed4215bb0bc38cd4364dca" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" is_pre_election: true
I20260620 19:13:02.870033 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a39aebaed4215bb0bc38cd4364dca" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1e6737a4894482a00e28e4baa20e1a" is_pre_election: true
I20260620 19:13:02.870144 26871 raft_consensus.cc:2454] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 0.
I20260620 19:13:02.870159 26734 raft_consensus.cc:2454] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 0.
I20260620 19:13:02.870285 26942 leader_election.cc:302] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12cc1ae5b92540dc91fb5ce15fe15f4e, 3aa0eeb2e8dc409b98093512e408b54c; no voters:
I20260620 19:13:02.870376 27149 raft_consensus.cc:2781] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:02.870416 27149 raft_consensus.cc:482] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:02.870433 27149 raft_consensus.cc:3037] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:02.871111 27149 raft_consensus.cc:504] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.871228 27149 leader_election.cc:288] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 election: Requested vote from peers 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997), 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777)
I20260620 19:13:02.871397 26734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a39aebaed4215bb0bc38cd4364dca" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1e6737a4894482a00e28e4baa20e1a"
I20260620 19:13:02.871411 26871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a13a39aebaed4215bb0bc38cd4364dca" candidate_uuid: "3aa0eeb2e8dc409b98093512e408b54c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e"
I20260620 19:13:02.871465 26734 raft_consensus.cc:3037] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:02.871475 26871 raft_consensus.cc:3037] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:02.872023 26734 raft_consensus.cc:2454] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 1.
I20260620 19:13:02.872078 26871 raft_consensus.cc:2454] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa0eeb2e8dc409b98093512e408b54c in term 1.
I20260620 19:13:02.872186 26941 leader_election.cc:302] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3aa0eeb2e8dc409b98093512e408b54c, 9c1e6737a4894482a00e28e4baa20e1a; no voters:
I20260620 19:13:02.872274 27149 raft_consensus.cc:2781] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:02.872421 27149 raft_consensus.cc:686] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 1 LEADER]: Becoming Leader. State: Replica: 3aa0eeb2e8dc409b98093512e408b54c, State: Running, Role: LEADER
I20260620 19:13:02.872478 27149 consensus_queue.cc:237] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } }
I20260620 19:13:02.872735 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.872627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:60099)
0620 19:13:02.872665 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.872701 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 19:13:02.873117 26414 catalog_manager.cc:5495] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195). New cstate: current_term: 1 leader_uuid: "3aa0eeb2e8dc409b98093512e408b54c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3aa0eeb2e8dc409b98093512e408b54c" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:02.891395 27139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.889457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.19.66.195:42635)
0620 19:13:02.889511 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:02.889513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:02.889520 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:02.889544 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:02.889544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:02.889552 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:02.889596 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:02.889699 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.890174 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.890873 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:02.891069 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:02.891114 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:02.891205 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:02.891228 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:02.891263 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:02.891297 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:13:02.891319 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:13:02.892234 26871 raft_consensus.cc:1261] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Refusing update from remote peer 3aa0eeb2e8dc409b98093512e408b54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:02.892257 26734 raft_consensus.cc:1261] T a13a39aebaed4215bb0bc38cd4364dca P 9c1e6737a4894482a00e28e4baa20e1a [term 1 FOLLOWER]: Refusing update from remote peer 3aa0eeb2e8dc409b98093512e408b54c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:02.892460 27149 consensus_queue.cc:1035] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12cc1ae5b92540dc91fb5ce15fe15f4e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:02.892568 27154 consensus_queue.cc:1035] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1e6737a4894482a00e28e4baa20e1a" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:02.893237 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.893129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:53829)
0620 19:13:02.893168 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.893204 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:13:02.893837 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.893742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:60219)
0620 19:13:02.893781 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.893806 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:13:02.894387 27151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.894251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.195:58101)
0620 19:13:02.894329 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.894359 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 19:13:02.894919 27153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.894827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.193:40401)
0620 19:13:02.894861 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.894885 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 19:13:02.895350 26414 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:02.895432 26414 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49386:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:02.895466 27147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:02.895385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46419 (local address 127.19.66.194:49507)
0620 19:13:02.895418 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.895440 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:46419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:13:02.896551 26414 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 10deb988d4764b8997e2005e3543510f: Sending DeleteTablet for 3 replicas of tablet a13a39aebaed4215bb0bc38cd4364dca
I20260620 19:13:02.896764 26714 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a39aebaed4215bb0bc38cd4364dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:02 UTC) from {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:60536
I20260620 19:13:02.896829 26851 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a39aebaed4215bb0bc38cd4364dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:02 UTC) from {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:56980
I20260620 19:13:02.896976 26713 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a39aebaed4215bb0bc38cd4364dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:02 UTC) from {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:60536
I20260620 19:13:02.897015 27168 tablet_replica.cc:330] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: stopping tablet replica
I20260620 19:13:02.897049 26987 tablet_service.cc:1514] Processing DeleteTablet for tablet a13a39aebaed4215bb0bc38cd4364dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:02 UTC) from {username='oryx', principal='oryx/127.19.66.253@KRBTEST.COM'} at 127.0.0.1:41796
I20260620 19:13:02.897065 27168 raft_consensus.cc:2229] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:02.897121 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26561
I20260620 19:13:02.897897 27169 tablet_replica.cc:330] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: stopping tablet replica
I20260620 19:13:02.897938 27169 raft_consensus.cc:2229] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:02.897979 27169 raft_consensus.cc:2258] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:02.898226 27169 ts_tablet_manager.cc:1905] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:02.898924 27168 raft_consensus.cc:2258] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:02.899243 27168 ts_tablet_manager.cc:1905] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:02.899709 27169 ts_tablet_manager.cc:1918] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:02.899766 27169 log.cc:1196] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/a13a39aebaed4215bb0bc38cd4364dca
I20260620 19:13:02.900005 27169 ts_tablet_manager.cc:1939] T a13a39aebaed4215bb0bc38cd4364dca P 3aa0eeb2e8dc409b98093512e408b54c: Deleting consensus metadata
I20260620 19:13:02.900282 26398 catalog_manager.cc:4841] TS 3aa0eeb2e8dc409b98093512e408b54c (127.19.66.195:42635): tablet a13a39aebaed4215bb0bc38cd4364dca (table test-table [id=9ea6b667848e415eba9dc550ad4c58a6]) successfully deleted
I20260620 19:13:02.900638 27168 ts_tablet_manager.cc:1918] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:02.900691 27168 log.cc:1196] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/a13a39aebaed4215bb0bc38cd4364dca
I20260620 19:13:02.900900 27168 ts_tablet_manager.cc:1939] T a13a39aebaed4215bb0bc38cd4364dca P 12cc1ae5b92540dc91fb5ce15fe15f4e: Deleting consensus metadata
I20260620 19:13:02.901186 26400 catalog_manager.cc:4841] TS 12cc1ae5b92540dc91fb5ce15fe15f4e (127.19.66.194:33997): tablet a13a39aebaed4215bb0bc38cd4364dca (table test-table [id=9ea6b667848e415eba9dc550ad4c58a6]) successfully deleted
W20260620 19:13:02.902097 27138 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:02.901983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41777 (local address 127.0.0.1:60552)
0620 19:13:02.902029 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:02.902055 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:41777: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:13:02.902208 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26788
W20260620 19:13:02.902231 26399 catalog_manager.cc:4584] TS 9c1e6737a4894482a00e28e4baa20e1a (127.19.66.193:41777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a13a39aebaed4215bb0bc38cd4364dca: Network error: Client connection negotiation failed: client connection to 127.19.66.193:41777: connect: Connection reset by peer (error 104)
I20260620 19:13:02.906531 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26924
I20260620 19:13:02.911528 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26274
I20260620 19:13:02.915643 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 26468
2026-06-20T19:13:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7830 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:02 dist-test-slave-nq8r krb5kdc[27172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:02 dist-test-slave-nq8r krb5kdc[27172](info): set up 2 sockets
Jun 20 19:13:02 dist-test-slave-nq8r krb5kdc[27172](info): commencing operation
krb5kdc: starting...
W20260620 19:13:03.856957 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:13:03 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab.
Entry for principal oryx/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab.
I20260620 19:13:03.961356 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35409
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab
--principal=oryx/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:04.044807 27185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:04.045040 27185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:04.045063 27185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:04.046181 27185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:04.046213 27185 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 19:13:04.046224 27185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:04.046236 27185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:04.046250 27185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:04.046260 27185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:04.047693 27185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35409
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.254.keytab
--principal=oryx/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:04.047870 27185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:04.048034 27185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:04.052469 27193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.052480 27190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.052476 27191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.052692 27185 server_base.cc:1034] running on GCE node
I20260620 19:13:04.052959 27185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:04.053232 27185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:04.054378 27185 hybrid_clock.cc:648] HybridClock initialized: now 1781982784054344 us; error 51 us; skew 500 ppm
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:04.056897 27185 init.cc:377] Logged in from keytab as oryx/127.19.66.254@KRBTEST.COM (short username oryx)
I20260620 19:13:04.057298 27185 webserver.cc:460] Webserver started at http://127.19.66.254:39193/ using document root <none> and password file <none>
I20260620 19:13:04.057492 27185 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:04.057534 27185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:04.057663 27185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:04.058693 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "9b77c89e327d4f6fb0532090c539a0d5"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "2cfb98f1d77952fe689c23066f120ee4"
server_key_iv: "b6d41b5a23de6f5c969853ae7c618e38"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.059026 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "9b77c89e327d4f6fb0532090c539a0d5"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "2cfb98f1d77952fe689c23066f120ee4"
server_key_iv: "b6d41b5a23de6f5c969853ae7c618e38"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.060266 27185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:13:04.060921 27200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.061064 27185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.061148 27185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "9b77c89e327d4f6fb0532090c539a0d5"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "2cfb98f1d77952fe689c23066f120ee4"
server_key_iv: "b6d41b5a23de6f5c969853ae7c618e38"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.061214 27185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:04.069284 27185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:04.071218 27185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:04.071349 27185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:04.075676 27185 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:35409
I20260620 19:13:04.075733 27262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:35409 every 8 connection(s)
I20260620 19:13:04.076028 27185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:04.076674 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.078922 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Bootstrap starting.
I20260620 19:13:04.079493 27263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.079753 27263 log.cc:824] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:04.080363 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: No bootstrap required, opened a new log
I20260620 19:13:04.082393 27263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } }
I20260620 19:13:04.082538 27263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.082561 27263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b77c89e327d4f6fb0532090c539a0d5, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.082690 27263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } }
I20260620 19:13:04.082754 27263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:04.082777 27263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:04.082823 27263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.083335 27263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } }
I20260620 19:13:04.083422 27263 leader_election.cc:302] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b77c89e327d4f6fb0532090c539a0d5; no voters:
I20260620 19:13:04.083572 27263 leader_election.cc:288] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:04.083634 27266 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:04.083802 27263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:04.083789 27266 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [term 1 LEADER]: Becoming Leader. State: Replica: 9b77c89e327d4f6fb0532090c539a0d5, State: Running, Role: LEADER
I20260620 19:13:04.083932 27266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } }
I20260620 19:13:04.084277 27267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b77c89e327d4f6fb0532090c539a0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } } }
I20260620 19:13:04.084362 27267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:04.084506 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b77c89e327d4f6fb0532090c539a0d5. Latest consensus state: current_term: 1 leader_uuid: "9b77c89e327d4f6fb0532090c539a0d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b77c89e327d4f6fb0532090c539a0d5" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35409 } } }
I20260620 19:13:04.084559 27274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:04.084736 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:04.085198 27274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:04.085268 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27185
I20260620 19:13:04.085371 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:13:04.085572 19723 external_mini_cluster.cc:1428] Setting key 06d1b2dbfd5378d442b6092c453824ce
I20260620 19:13:04.086654 27274 catalog_manager.cc:1347] Generated new cluster ID: 096c6c2d664047d691e58bb05be74793
I20260620 19:13:04.086704 27274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:04.094348 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.085971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37854 (local address 127.19.66.254:35409)
0620 19:13:04.086159 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.086167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.086198 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:13:04.086310 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.086316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.086372 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.086517 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.087178 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.088069 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.088776 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.089267 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.089360 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.091487 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.091491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.091491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.091504 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.093001 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.093302 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.093305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.093306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.093386 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.093551 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.093552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.093552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.093771 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.093913 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.093983 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.094191 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260620 19:13:04.097126 27274 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:04.097815 27274 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:04.107049 27274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Generated new TSK 0
I20260620 19:13:04.107227 27274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
I20260620 19:13:04.135336 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35409
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:04.213047 27289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:04.213244 27289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:04.213274 27289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:04.214408 27289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:04.214447 27289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:04.214473 27289 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:13:04.214496 27289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:04.216042 27289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:04.216269 27289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:04.216481 27289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:04.221129 27295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.221151 27297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.221164 27294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.221215 27289 server_base.cc:1034] running on GCE node
I20260620 19:13:04.221510 27289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:04.221711 27289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:04.222838 27289 hybrid_clock.cc:648] HybridClock initialized: now 1781982784222827 us; error 30 us; skew 500 ppm
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:04.225152 27289 init.cc:377] Logged in from keytab as oryx/127.19.66.193@KRBTEST.COM (short username oryx)
I20260620 19:13:04.225534 27289 webserver.cc:460] Webserver started at http://127.19.66.193:40857/ using document root <none> and password file <none>
I20260620 19:13:04.225734 27289 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:04.225780 27289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:04.225899 27289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:04.226872 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "b008df4d5c6efa4bef93c6a9f9bc5ba1"
server_key_iv: "515199da8acbf5d00588b9316620422f"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.227190 27289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "b008df4d5c6efa4bef93c6a9f9bc5ba1"
server_key_iv: "515199da8acbf5d00588b9316620422f"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.228340 27289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:13:04.228965 27304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.229100 27289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:04.229177 27289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "b008df4d5c6efa4bef93c6a9f9bc5ba1"
server_key_iv: "515199da8acbf5d00588b9316620422f"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.229239 27289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:04.250584 27289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:04.252282 27289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:04.252413 27289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:04.252921 27289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:04.252961 27289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.252991 27289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:04.253015 27289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:04.259291 27289 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:39121
I20260620 19:13:04.259361 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:39121 every 8 connection(s)
I20260620 19:13:04.259671 27289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:04.259845 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27289
I20260620 19:13:04.259948 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:04.260169 19723 external_mini_cluster.cc:1428] Setting key 9a22f5677644d061c5b9ec83d396718b
I20260620 19:13:04.262598 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.253335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:39315 (local address 127.19.66.254:35409)
0620 19:13:04.253375 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.253377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.253587 (+ 210us) server_negotiation.cc:407] Connection header received
0620 19:13:04.254094 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.254095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.254107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.254156 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.254804 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.255558 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.256996 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.257219 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.257275 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.259271 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.259273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.259274 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.259282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.261371 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.261736 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.261737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.261738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.261797 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.261986 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.261987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.261987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.262124 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.262191 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.262468 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.262534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:13:04.262840 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.253346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.193:39315)
0620 19:13:04.253479 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.253512 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.253986 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.254222 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.254226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.254490 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.254720 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.254725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.255640 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.255641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.256912 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.256915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.257304 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.257304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.257431 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.257891 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.257900 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.259183 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.261450 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.261455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.261457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.261671 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.261851 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.261852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.261852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.261937 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.262184 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.262189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.262345 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.262452 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.262623 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260620 19:13:04.263373 27420 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35409
I20260620 19:13:04.263489 27420 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:04.263695 27420 heartbeater.cc:511] Master 127.19.66.254:35409 requested a full tablet report, sending...
I20260620 19:13:04.264205 27216 ts_manager.cc:194] Registered new tserver with Master: b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121)
I20260620 19:13:04.265025 27216 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:39315
WARNING: no policy specified for oryx/127.19.66.194@KRBTEST.COM; defaulting to no policy
I20260620 19:13:04.269995 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.264410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.193:34753)
0620 19:13:04.264557 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.264568 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.264626 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.264756 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.264757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.264972 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.265032 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.265033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.265946 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.265947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.266738 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.266741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.267056 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.267057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.267123 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.267485 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.267490 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.267831 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.269303 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.269304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.269304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.269452 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.269604 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.269604 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.269604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.269646 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.269792 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.269793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.269824 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.269865 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.269930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260620 19:13:04.270118 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.264365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:34753 (local address 127.19.66.254:35409)
0620 19:13:04.264469 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.264471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.264625 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:13:04.264686 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.264687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.264697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.264752 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.265165 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.265844 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.266811 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.266995 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.267037 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.267905 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.267906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.267906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.267911 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.269222 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.269505 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.269506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.269506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.269553 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.269678 (+ 125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.269678 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.269679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.269761 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.269800 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.269880 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.270027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
I20260620 19:13:04.282665 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35409
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:04.361063 27429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:04.361255 27429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:04.361276 27429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:04.362391 27429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:04.362434 27429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:04.362455 27429 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:13:04.362484 27429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:04.364003 27429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:04.364224 27429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:04.364429 27429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:04.368970 27437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.368975 27435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.369076 27429 server_base.cc:1034] running on GCE node
W20260620 19:13:04.368980 27434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.369405 27429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:04.369642 27429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:04.370779 27429 hybrid_clock.cc:648] HybridClock initialized: now 1781982784370765 us; error 27 us; skew 500 ppm
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:04.373099 27429 init.cc:377] Logged in from keytab as oryx/127.19.66.194@KRBTEST.COM (short username oryx)
I20260620 19:13:04.373464 27429 webserver.cc:460] Webserver started at http://127.19.66.194:45133/ using document root <none> and password file <none>
I20260620 19:13:04.373653 27429 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:04.373693 27429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:04.373785 27429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:04.374763 27429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "372c27a542774b9bbb8db28e4548c7e6"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "4ed142c6d9c75c5f9d1a8a44fa31621e"
server_key_iv: "f99f38ed853cdb48cb25495461bcd0b6"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.375043 27429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "372c27a542774b9bbb8db28e4548c7e6"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "4ed142c6d9c75c5f9d1a8a44fa31621e"
server_key_iv: "f99f38ed853cdb48cb25495461bcd0b6"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.376194 27429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:13:04.376936 27444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.377071 27429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.377146 27429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "372c27a542774b9bbb8db28e4548c7e6"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "4ed142c6d9c75c5f9d1a8a44fa31621e"
server_key_iv: "f99f38ed853cdb48cb25495461bcd0b6"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.377197 27429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:04.386266 27429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:04.387949 27429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:04.388051 27429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:04.388468 27429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:04.388502 27429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.388522 27429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:04.388532 27429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.394364 27429 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:36927
I20260620 19:13:04.394495 27559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:36927 every 8 connection(s)
I20260620 19:13:04.394726 27429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:04.396823 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27429
I20260620 19:13:04.396955 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:04.397135 19723 external_mini_cluster.cc:1428] Setting key 64fb68ecf3ed7675b730a06ed01b4834
I20260620 19:13:04.397614 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.388897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:55853 (local address 127.19.66.254:35409)
0620 19:13:04.388961 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.388963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.389123 (+ 160us) server_negotiation.cc:407] Connection header received
0620 19:13:04.389589 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.389590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.389603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.389681 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.390424 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.391164 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.392599 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.392913 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.392980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.395237 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.395238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.395238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.395245 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.396289 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.396614 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.396614 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.396614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.396655 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.396925 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.396926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.396927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.397081 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.397141 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.397444 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.397525 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:13:04.397716 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.388868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.194:55853)
0620 19:13:04.389003 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.389042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.389504 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.389752 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.389755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.390060 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.390316 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.390322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.391256 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.391257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.392504 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.392508 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.392995 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.392995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.393097 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.393602 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.393611 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.395160 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.396348 (+ 1188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.396353 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.396355 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.396556 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.396706 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.396706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.396706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.396831 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.397145 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.397151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.397311 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.397417 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.397545 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260620 19:13:04.398169 27560 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35409
I20260620 19:13:04.398255 27560 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:04.398429 27560 heartbeater.cc:511] Master 127.19.66.254:35409 requested a full tablet report, sending...
I20260620 19:13:04.398797 27216 ts_manager.cc:194] Registered new tserver with Master: 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927)
I20260620 19:13:04.399312 27216 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:55853
WARNING: no policy specified for oryx/127.19.66.195@KRBTEST.COM; defaulting to no policy
I20260620 19:13:04.405498 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.400058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.194:42475)
0620 19:13:04.400206 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.400215 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.400264 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.400370 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.400370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.400505 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.400550 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.400551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.401383 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.401383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.401921 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.401923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.402343 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.402343 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.402385 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.402796 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.402800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.403036 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.404530 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.404530 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.404531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.404685 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.404929 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.404929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.404930 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.404977 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.405271 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.405271 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.405313 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.405366 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.405434 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260620 19:13:04.405550 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.400113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:42475 (local address 127.19.66.254:35409)
0620 19:13:04.400150 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.400152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.400273 (+ 121us) server_negotiation.cc:407] Connection header received
0620 19:13:04.400297 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.400298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.400309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.400367 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.400609 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.401308 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.401998 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.402276 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.402347 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.403114 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.403115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.403115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.403122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.404442 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.404755 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.404756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.404757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.404810 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.405029 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.405030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.405030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.405212 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.405282 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.405395 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.405463 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
I20260620 19:13:04.419414 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35409
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:04.497100 27569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:04.497298 27569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:04.497319 27569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:04.498409 27569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:04.498440 27569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:04.498457 27569 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 19:13:04.498473 27569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:04.499972 27569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.19.66.254:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:04.500172 27569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:04.500344 27569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:04.504946 27577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.505023 27569 server_base.cc:1034] running on GCE node
W20260620 19:13:04.504920 27575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:04.504940 27574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:04.505309 27569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:04.505553 27569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:04.506690 27569 hybrid_clock.cc:648] HybridClock initialized: now 1781982784506678 us; error 35 us; skew 500 ppm
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:04.509040 27569 init.cc:377] Logged in from keytab as oryx/127.19.66.195@KRBTEST.COM (short username oryx)
I20260620 19:13:04.509445 27569 webserver.cc:460] Webserver started at http://127.19.66.195:35667/ using document root <none> and password file <none>
I20260620 19:13:04.509647 27569 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:04.509694 27569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:04.509817 27569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:04.510872 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "48e44646adff4e2d9ba9896c5d1659b4"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "bf0dfa6b205c66c868856031d70c996c"
server_key_iv: "3e6b8686d60495e13ac8e5689025327c"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.511193 27569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "48e44646adff4e2d9ba9896c5d1659b4"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "bf0dfa6b205c66c868856031d70c996c"
server_key_iv: "3e6b8686d60495e13ac8e5689025327c"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.512388 27569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:04.513116 27584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.513253 27569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.513334 27569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "48e44646adff4e2d9ba9896c5d1659b4"
format_stamp: "Formatted at 2026-06-20 19:13:04 on dist-test-slave-nq8r"
server_key: "bf0dfa6b205c66c868856031d70c996c"
server_key_iv: "3e6b8686d60495e13ac8e5689025327c"
server_key_version: "encryptionkey@0"
I20260620 19:13:04.513389 27569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:04.527812 27569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:04.529552 27569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:04.529680 27569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:04.530138 27569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:04.530179 27569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.530208 27569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:04.530232 27569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.536681 27569 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:34225
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:04.536809 27699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:34225 every 8 connection(s)
I20260620 19:13:04.537070 27569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:04.539753 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.530577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:49459 (local address 127.19.66.254:35409)
0620 19:13:04.530624 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.530626 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.530830 (+ 204us) server_negotiation.cc:407] Connection header received
0620 19:13:04.531328 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.531329 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.531342 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.531394 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.532206 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.532981 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.534478 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.534775 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.534845 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.537294 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.537295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.537295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.537303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.538502 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.538868 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.538869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.538869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.538928 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.539153 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.539154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.539154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.539268 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.539323 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.539578 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.539666 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:13:04.539903 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.530579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.195:49459)
0620 19:13:04.530716 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.530750 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.531231 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.531495 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.531500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.531922 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.532114 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.532119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.533096 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.533097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.534387 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.534392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.534864 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.534865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.535014 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.535506 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.535516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.537206 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.538589 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.538596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.538598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.538802 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.538979 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.538980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.538980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.539097 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.539322 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.539328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.539463 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.539547 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.539694 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260620 19:13:04.540330 27700 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35409
I20260620 19:13:04.540410 27700 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:04.540572 27700 heartbeater.cc:511] Master 127.19.66.254:35409 requested a full tablet report, sending...
I20260620 19:13:04.540944 27217 ts_manager.cc:194] Registered new tserver with Master: 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:04.541306 27217 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:49459
I20260620 19:13:04.543692 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27569
I20260620 19:13:04.543798 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:04.543964 19723 external_mini_cluster.cc:1428] Setting key 9527d0410a764ce242af4a1bfd26b346
I20260620 19:13:04.544838 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:04.546772 27706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.541568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.19.66.195:49319)
0620 19:13:04.541719 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.541729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.541784 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.541888 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.541889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.542063 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:04.542135 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.542137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.542719 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.542719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.543214 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.543215 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.543519 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.543521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.543605 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.544011 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:04.544016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:04.544342 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:04.545840 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.545841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.545842 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.546029 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.546219 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:04.546219 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:04.546220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:04.546278 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:04.546531 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:04.546532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:04.546586 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.546637 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.546710 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260620 19:13:04.546828 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.541625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:49319 (local address 127.19.66.254:35409)
0620 19:13:04.541665 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.541667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.541774 (+ 107us) server_negotiation.cc:407] Connection header received
0620 19:13:04.541834 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.541834 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.541843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.541884 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.542193 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.542654 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.543263 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.543426 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.543466 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.544438 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.544439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.544439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.544446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.545765 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.546108 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.546109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.546109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.546163 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.546328 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.546328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.546329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.546443 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.546538 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.546671 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.546742 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:13:04.565377 27285 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:04.560053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.19.66.254:35409)
0620 19:13:04.560118 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.560120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.560134 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:13:04.560215 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.560216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.560226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.560274 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.560814 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.561342 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.562046 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.562244 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.562299 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.565289 (+ 2990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37856: BlockingRecv error: recv got EOF from 127.0.0.1:37856 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 19:13:04 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:04.571942 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.566035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.19.66.254:35409)
0620 19:13:04.566083 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.566085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.566188 (+ 103us) server_negotiation.cc:407] Connection header received
0620 19:13:04.566225 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.566225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.566234 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.566280 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:04.566654 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.567120 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.567730 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.567894 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.567934 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.569818 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:04.569819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:04.569819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:04.569827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:04.571016 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.571301 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.571302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.571303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.571349 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:04.571506 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:04.571506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:04.571507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:04.571608 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:04.571691 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.571782 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.571853 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:04.573287 27217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:04.574823 27217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:04.580902 27721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36927 (local address 127.0.0.1:38614)
0620 19:13:04.576943 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.576984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.577045 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.577350 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.577352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.577355 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.577440 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.577442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.578743 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.578744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.579820 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.579823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.580586 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.580587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.580664 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.580680 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.580750 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.580802 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":69,"thread_start_us":83,"threads_started":1}
I20260620 19:13:04.580915 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.19.66.194:36927)
0620 19:13:04.577212 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.577216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.577225 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:04.577258 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.577259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.577273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.577345 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.577524 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.578678 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.579918 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.580507 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.580570 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.580680 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.580765 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.580801 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":68,"threads_started":1}
I20260620 19:13:04.581125 27720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:34225 (local address 127.0.0.1:32882)
0620 19:13:04.577146 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.577154 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.577186 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.577655 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.577656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.577658 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.577700 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.577702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.578529 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.578530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.579839 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.579842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.580893 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.580894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.580955 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.580959 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.581015 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.581052 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":65,"threads_started":1}
I20260620 19:13:04.581414 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.19.66.195:34225)
0620 19:13:04.577457 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.577461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.577470 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:04.577512 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.577514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.577529 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.577594 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.577770 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.578450 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.579929 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.580711 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.580765 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.580835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.581284 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.581324 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":65,"threads_started":1}
I20260620 19:13:04.581781 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:39121 (local address 127.0.0.1:59236)
0620 19:13:04.577095 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.577104 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.577144 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.577350 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.577352 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.577357 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.577446 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.577447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.579555 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.579556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.580569 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.580571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.581566 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.581567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.581625 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.581629 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.581679 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.581717 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260620 19:13:04.581830 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.576932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.19.66.193:39121)
0620 19:13:04.577123 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.577128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.577139 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:13:04.577233 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.577236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.577252 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.577341 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.577525 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.578664 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.580664 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.581393 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.581450 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.581533 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.581695 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.581729 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
I20260620 19:13:04.581849 27494 tablet_service.cc:1467] Processing CreateTablet for tablet 18120bcaff4a4e45891882614205d60c (DEFAULT_TABLE table=test-table [id=042016b710b246f7a2d08be66e1d032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.581984 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 18120bcaff4a4e45891882614205d60c (DEFAULT_TABLE table=test-table [id=042016b710b246f7a2d08be66e1d032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.582221 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18120bcaff4a4e45891882614205d60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.582237 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18120bcaff4a4e45891882614205d60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.585851 27725 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Bootstrap starting.
I20260620 19:13:04.586051 27726 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Bootstrap starting.
I20260620 19:13:04.586567 27725 tablet_bootstrap.cc:654] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.586568 27726 tablet_bootstrap.cc:654] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.586585 27354 tablet_service.cc:1467] Processing CreateTablet for tablet 18120bcaff4a4e45891882614205d60c (DEFAULT_TABLE table=test-table [id=042016b710b246f7a2d08be66e1d032f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.586900 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18120bcaff4a4e45891882614205d60c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.586887 27726 log.cc:824] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:04.586887 27725 log.cc:824] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:04.587600 27726 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: No bootstrap required, opened a new log
I20260620 19:13:04.587672 27726 ts_tablet_manager.cc:1397] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:04.587815 27725 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: No bootstrap required, opened a new log
I20260620 19:13:04.587877 27725 ts_tablet_manager.cc:1397] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:04.589654 27726 raft_consensus.cc:348] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.589740 27725 raft_consensus.cc:348] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.589829 27726 raft_consensus.cc:374] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.589852 27725 raft_consensus.cc:374] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.589864 27726 raft_consensus.cc:729] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.589882 27725 raft_consensus.cc:729] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.590008 27726 consensus_queue.cc:260] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.590004 27725 consensus_queue.cc:260] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.590260 27726 ts_tablet_manager.cc:1428] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:13:04.590260 27725 ts_tablet_manager.cc:1428] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:13:04.590349 27700 heartbeater.cc:503] Master 127.19.66.254:35409 was elected leader, sending a full tablet report...
I20260620 19:13:04.590389 27729 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Bootstrap starting.
I20260620 19:13:04.590680 27560 heartbeater.cc:503] Master 127.19.66.254:35409 was elected leader, sending a full tablet report...
I20260620 19:13:04.590996 27729 tablet_bootstrap.cc:654] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.591290 27729 log.cc:824] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:04.591892 27729 tablet_bootstrap.cc:492] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: No bootstrap required, opened a new log
I20260620 19:13:04.591950 27729 ts_tablet_manager.cc:1397] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:04.593408 27729 raft_consensus.cc:348] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.593537 27729 raft_consensus.cc:374] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.593566 27729 raft_consensus.cc:729] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.593688 27729 consensus_queue.cc:260] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.593918 27729 ts_tablet_manager.cc:1428] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:04.593962 27420 heartbeater.cc:503] Master 127.19.66.254:35409 was elected leader, sending a full tablet report...
W20260620 19:13:04.645228 27561 tablet.cc:2369] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:13:04.760445 27421 tablet.cc:2369] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:13:04.787635 27701 tablet.cc:2369] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:04.871922 27731 raft_consensus.cc:482] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:04.872040 27731 raft_consensus.cc:504] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.872429 27731 leader_election.cc:288] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927), b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121)
I20260620 19:13:04.875167 27722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.872677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:60441 (local address 127.19.66.194:36927)
0620 19:13:04.872732 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.872734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.872954 (+ 220us) server_negotiation.cc:407] Connection header received
0620 19:13:04.873009 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.873011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.873020 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.873065 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.873191 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.873652 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874523 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.874862 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874921 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.874929 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.875065 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.875108 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:13:04.875180 27734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.872643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36927 (local address 127.19.66.195:60441)
0620 19:13:04.872885 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.872893 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.872956 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.873067 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.873068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.873069 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.873135 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.873137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.873720 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.873721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.874438 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.874440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.874933 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.875003 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.875006 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.875049 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.875087 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":70,"thread_start_us":93,"threads_started":1}
I20260620 19:13:04.875461 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.872642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:39121 (local address 127.19.66.195:60067)
0620 19:13:04.872715 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.872726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.872772 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.872922 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.872923 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.872925 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.873008 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.873010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.873598 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.873599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.874367 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.874369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.874867 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.875298 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.875303 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.875355 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.875393 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":25}
I20260620 19:13:04.875478 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.872672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:60067 (local address 127.19.66.193:39121)
0620 19:13:04.872723 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.872726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.872756 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:13:04.872839 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.872840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.872873 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.872920 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.873065 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.873532 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874449 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.874802 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.874862 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.874871 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.875380 (+ 509us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.875420 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:13:04.875516 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18120bcaff4a4e45891882614205d60c" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6" is_pre_election: true
I20260620 19:13:04.875684 27514 raft_consensus.cc:2454] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 0.
I20260620 19:13:04.875756 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18120bcaff4a4e45891882614205d60c" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" is_pre_election: true
I20260620 19:13:04.875897 27374 raft_consensus.cc:2454] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 0.
I20260620 19:13:04.875917 27586 leader_election.cc:302] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:04.876030 27731 raft_consensus.cc:2781] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:04.876099 27731 raft_consensus.cc:482] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:04.876122 27731 raft_consensus.cc:3037] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.876784 27731 raft_consensus.cc:504] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.876955 27731 leader_election.cc:288] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 election: Requested vote from peers 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927), b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121)
I20260620 19:13:04.877101 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18120bcaff4a4e45891882614205d60c" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
I20260620 19:13:04.877101 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18120bcaff4a4e45891882614205d60c" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6"
I20260620 19:13:04.877179 27374 raft_consensus.cc:3037] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.877179 27514 raft_consensus.cc:3037] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.878010 27514 raft_consensus.cc:2454] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 1.
I20260620 19:13:04.878007 27374 raft_consensus.cc:2454] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 1.
I20260620 19:13:04.878162 27586 leader_election.cc:302] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:04.878237 27731 raft_consensus.cc:2781] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:04.878365 27731 raft_consensus.cc:686] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 LEADER]: Becoming Leader. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Running, Role: LEADER
I20260620 19:13:04.878455 27731 consensus_queue.cc:237] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } }
I20260620 19:13:04.879259 27216 catalog_manager.cc:5495] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195). New cstate: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:04.939121 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.937061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.19.66.195:34225)
0620 19:13:04.937113 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.937115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.937122 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:04.937158 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.937158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.937174 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.937234 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:04.937348 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.937827 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.938508 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.938694 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.938749 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.938862 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:04.938960 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:04.939014 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.939030 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.939047 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:13:04.940917 27374 raft_consensus.cc:1261] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Refusing update from remote peer 48e44646adff4e2d9ba9896c5d1659b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:04.940917 27514 raft_consensus.cc:1261] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Refusing update from remote peer 48e44646adff4e2d9ba9896c5d1659b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:04.941221 27731 consensus_queue.cc:1035] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:04.941357 27735 consensus_queue.cc:1035] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:04.943653 27739 mvcc.cc:205] Tried to move back new op lower bound from 7299001487115018240 to 7299001486862712832. Current Snapshot: MvccSnapshot[applied={T|T < 7299001487115018240}]
I20260620 19:13:04.945580 27216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:04.945669 27216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:04.946359 27216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Sending DeleteTablet for 3 replicas of tablet 18120bcaff4a4e45891882614205d60c
I20260620 19:13:04.946632 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet 18120bcaff4a4e45891882614205d60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:04 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:38614
I20260620 19:13:04.946674 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 18120bcaff4a4e45891882614205d60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:04 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:32882
I20260620 19:13:04.946784 27354 tablet_service.cc:1514] Processing DeleteTablet for tablet 18120bcaff4a4e45891882614205d60c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:04 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:59236
I20260620 19:13:04.946820 27747 tablet_replica.cc:330] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: stopping tablet replica
I20260620 19:13:04.946908 27748 tablet_replica.cc:330] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: stopping tablet replica
I20260620 19:13:04.946987 27748 raft_consensus.cc:2229] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:04.946987 27747 raft_consensus.cc:2229] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:04.947077 27748 raft_consensus.cc:2258] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:04.947165 27747 raft_consensus.cc:2258] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:04.947177 27216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:04.947307 27216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:04.947523 27748 ts_tablet_manager.cc:1905] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:04.947801 27747 ts_tablet_manager.cc:1905] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:04.948354 27494 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6e4275c636418ba47833d72d059430 (DEFAULT_TABLE table=test-table [id=edd460eed1df43fdb822f2a0488780d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.948474 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6e4275c636418ba47833d72d059430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.949137 27748 ts_tablet_manager.cc:1918] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:04.949203 27748 log.cc:1196] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/18120bcaff4a4e45891882614205d60c
I20260620 19:13:04.949446 27748 ts_tablet_manager.cc:1939] T 18120bcaff4a4e45891882614205d60c P 48e44646adff4e2d9ba9896c5d1659b4: Deleting consensus metadata
I20260620 19:13:04.949707 27747 ts_tablet_manager.cc:1918] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:04.949749 27726 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Bootstrap starting.
I20260620 19:13:04.949771 27747 log.cc:1196] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/18120bcaff4a4e45891882614205d60c
I20260620 19:13:04.949978 27747 ts_tablet_manager.cc:1939] T 18120bcaff4a4e45891882614205d60c P 372c27a542774b9bbb8db28e4548c7e6: Deleting consensus metadata
I20260620 19:13:04.950274 27749 tablet_replica.cc:330] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: stopping tablet replica
I20260620 19:13:04.950330 27726 tablet_bootstrap.cc:654] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.950367 27749 raft_consensus.cc:2229] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:04.950456 27749 raft_consensus.cc:2258] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:04.950714 27202 catalog_manager.cc:4841] TS 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927): tablet 18120bcaff4a4e45891882614205d60c (table test-table [id=042016b710b246f7a2d08be66e1d032f]) successfully deleted
I20260620 19:13:04.950788 27749 ts_tablet_manager.cc:1905] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:04.950934 27204 catalog_manager.cc:4841] TS 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225): tablet 18120bcaff4a4e45891882614205d60c (table test-table [id=042016b710b246f7a2d08be66e1d032f]) successfully deleted
I20260620 19:13:04.950968 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6e4275c636418ba47833d72d059430 (DEFAULT_TABLE table=test-table [id=edd460eed1df43fdb822f2a0488780d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.951045 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6e4275c636418ba47833d72d059430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.951287 27354 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6e4275c636418ba47833d72d059430 (DEFAULT_TABLE table=test-table [id=edd460eed1df43fdb822f2a0488780d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:04.951381 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6e4275c636418ba47833d72d059430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:04.952107 27725 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Bootstrap starting.
I20260620 19:13:04.952265 27749 ts_tablet_manager.cc:1918] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:04.952322 27749 log.cc:1196] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/18120bcaff4a4e45891882614205d60c
I20260620 19:13:04.952479 27725 tablet_bootstrap.cc:654] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.952529 27749 ts_tablet_manager.cc:1939] T 18120bcaff4a4e45891882614205d60c P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting consensus metadata
I20260620 19:13:04.952582 27729 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Bootstrap starting.
I20260620 19:13:04.952947 27729 tablet_bootstrap.cc:654] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:04.953152 27725 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: No bootstrap required, opened a new log
I20260620 19:13:04.953191 27725 ts_tablet_manager.cc:1397] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:04.953317 27725 raft_consensus.cc:348] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.953367 27725 raft_consensus.cc:374] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.953384 27725 raft_consensus.cc:729] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.953423 27725 consensus_queue.cc:260] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.953433 27203 catalog_manager.cc:4841] TS b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121): tablet 18120bcaff4a4e45891882614205d60c (table test-table [id=042016b710b246f7a2d08be66e1d032f]) successfully deleted
I20260620 19:13:04.953502 27725 ts_tablet_manager.cc:1428] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.953971 27729 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: No bootstrap required, opened a new log
I20260620 19:13:04.954030 27729 ts_tablet_manager.cc:1397] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:13:04.954155 27729 raft_consensus.cc:348] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.954219 27729 raft_consensus.cc:374] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.954245 27729 raft_consensus.cc:729] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.954289 27729 consensus_queue.cc:260] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.954389 27729 ts_tablet_manager.cc:1428] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.954690 27726 tablet_bootstrap.cc:492] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: No bootstrap required, opened a new log
I20260620 19:13:04.954737 27726 ts_tablet_manager.cc:1397] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:13:04.954867 27726 raft_consensus.cc:348] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.954950 27726 raft_consensus.cc:374] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:04.954970 27726 raft_consensus.cc:729] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Initialized, Role: FOLLOWER
I20260620 19:13:04.955019 27726 consensus_queue.cc:260] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.955113 27726 ts_tablet_manager.cc:1428] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:04.979943 27730 raft_consensus.cc:482] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:04.980038 27730 raft_consensus.cc:504] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.980386 27730 leader_election.cc:288] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:04.984293 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.980683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:43193 (local address 127.19.66.195:34225)
0620 19:13:04.980725 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.980727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.981163 (+ 436us) server_negotiation.cc:407] Connection header received
0620 19:13:04.981259 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.981260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.981271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.981345 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.981527 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.982257 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.983387 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.983880 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.983948 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.983957 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.984177 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.984220 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:04.984314 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.980557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:34225 (local address 127.19.66.194:43193)
0620 19:13:04.981106 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.981115 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.981174 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.981374 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.981375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.981377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.981446 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.981448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.982349 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.982350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.983317 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.983319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.984028 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.984029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.984106 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.984109 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.984158 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.984194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":1809,"threads_started":1}
I20260620 19:13:04.984586 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6e4275c636418ba47833d72d059430" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4" is_pre_election: true
I20260620 19:13:04.984730 27654 raft_consensus.cc:2454] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 0.
I20260620 19:13:04.984956 27448 leader_election.cc:302] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:04.985296 27730 raft_consensus.cc:2781] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:04.985343 27730 raft_consensus.cc:482] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:04.985363 27730 raft_consensus.cc:3037] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.985548 27723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.980996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:48229 (local address 127.19.66.193:39121)
0620 19:13:04.981063 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:04.981064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:04.982485 (+ 1421us) server_negotiation.cc:407] Connection header received
0620 19:13:04.982590 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:04.982592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:04.982602 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:04.982664 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:04.982790 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.983552 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.984687 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.985193 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.985255 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.985263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:04.985454 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 19:13:04.985491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:13:04.985574 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:04.980939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:39121 (local address 127.19.66.194:48229)
0620 19:13:04.982407 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:04.982417 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:04.982512 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:04.982672 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:04.982672 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:04.982674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:04.982730 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.982731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.983636 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.983637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.984586 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:04.984588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:04.985277 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:04.985278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:04.985358 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:04.985362 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:04.985441 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:13:04.985479 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1453,"thread_start_us":108,"threads_started":1}
I20260620 19:13:04.986123 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6e4275c636418ba47833d72d059430" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" is_pre_election: true
I20260620 19:13:04.986176 27730 raft_consensus.cc:504] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.986225 27374 raft_consensus.cc:2454] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 0.
I20260620 19:13:04.986289 27730 leader_election.cc:288] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 election: Requested vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:04.986407 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6e4275c636418ba47833d72d059430" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
I20260620 19:13:04.986454 27374 raft_consensus.cc:3037] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.987176 27374 raft_consensus.cc:2454] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 1.
I20260620 19:13:04.987275 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6e4275c636418ba47833d72d059430" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4"
I20260620 19:13:04.987346 27654 raft_consensus.cc:3037] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:04.988185 27654 raft_consensus.cc:2454] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 1.
I20260620 19:13:04.988329 27448 leader_election.cc:302] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:04.988422 27730 raft_consensus.cc:2781] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:04.988541 27730 raft_consensus.cc:686] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Running, Role: LEADER
I20260620 19:13:04.988618 27730 consensus_queue.cc:237] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:04.989317 27217 catalog_manager.cc:5495] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "372c27a542774b9bbb8db28e4548c7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } health_report { overall_health: UNKNOWN } } }
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982784, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.254@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:05.031718 27285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.021570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.19.66.254:35409)
0620 19:13:05.021626 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.021627 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.021711 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:13:05.021784 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.021784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.021793 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.021841 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:05.022449 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.022925 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.023922 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.024089 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.024131 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.029362 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:05.029364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:05.029364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:05.029370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:05.030632 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:05.030942 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:05.030943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:05.030943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:05.030989 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:05.031209 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:05.031210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:05.031210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:05.031319 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:05.031376 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.031509 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.031613 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":39}
I20260620 19:13:05.031791 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.021472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35409 (local address 127.0.0.1:37874)
0620 19:13:05.021629 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.021642 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.021725 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.021868 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.021869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.022315 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:05.022377 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.022380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.023002 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.023003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.023850 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.023853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.024165 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.024166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.024238 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.027756 (+ 3518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:05.027762 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:05.029287 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:05.030700 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:05.030706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:05.030708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:05.030881 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:05.031040 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:05.031041 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:05.031041 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:05.031127 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:05.031372 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:05.031379 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:05.031426 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.031485 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.031679 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":92,"thread_start_us":67,"threads_started":1}
I20260620 19:13:05.033689 27217 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:37874:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 19:13:05.033864 27217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:05.035233 27354 tablet_service.cc:1467] Processing CreateTablet for tablet 66d621d98ad94cd2b9e362d64ecd2418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4d7f3c8cf03444b8ea79b8a04c1f5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:13:05.035291 27634 tablet_service.cc:1467] Processing CreateTablet for tablet 66d621d98ad94cd2b9e362d64ecd2418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4d7f3c8cf03444b8ea79b8a04c1f5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:13:05.035318 27494 tablet_service.cc:1467] Processing CreateTablet for tablet 66d621d98ad94cd2b9e362d64ecd2418 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f4d7f3c8cf03444b8ea79b8a04c1f5a7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 19:13:05.035385 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d621d98ad94cd2b9e362d64ecd2418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.035430 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d621d98ad94cd2b9e362d64ecd2418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.035446 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d621d98ad94cd2b9e362d64ecd2418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.036814 27729 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: Bootstrap starting.
I20260620 19:13:05.036932 27726 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: Bootstrap starting.
I20260620 19:13:05.037178 27729 tablet_bootstrap.cc:654] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.037326 27726 tablet_bootstrap.cc:654] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.037443 27725 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: Bootstrap starting.
I20260620 19:13:05.037818 27725 tablet_bootstrap.cc:654] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.038159 27726 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: No bootstrap required, opened a new log
I20260620 19:13:05.038208 27726 ts_tablet_manager.cc:1397] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:05.038337 27726 raft_consensus.cc:348] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.038403 27726 raft_consensus.cc:374] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.038421 27726 raft_consensus.cc:729] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.038442 27729 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: No bootstrap required, opened a new log
I20260620 19:13:05.038478 27729 ts_tablet_manager.cc:1397] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:13:05.038466 27726 consensus_queue.cc:260] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.038552 27726 ts_tablet_manager.cc:1428] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.038618 27729 raft_consensus.cc:348] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.038601 27730 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.038673 27729 raft_consensus.cc:374] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.038691 27729 raft_consensus.cc:729] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.038679 27730 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.038729 27729 consensus_queue.cc:260] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.038815 27729 ts_tablet_manager.cc:1428] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.038856 27733 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.038929 27733 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.039427 27725 tablet_bootstrap.cc:492] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: No bootstrap required, opened a new log
I20260620 19:13:05.039469 27725 ts_tablet_manager.cc:1397] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:05.039592 27725 raft_consensus.cc:348] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.039640 27725 raft_consensus.cc:374] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.039652 27725 raft_consensus.cc:729] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.039682 27725 consensus_queue.cc:260] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.039762 27725 ts_tablet_manager.cc:1428] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.039772 27735 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.039867 27735 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.436808 27730 consensus_queue.cc:1035] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.438220 27735 raft_consensus.cc:482] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:05.438290 27735 raft_consensus.cc:504] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.438400 27735 leader_election.cc:288] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927)
I20260620 19:13:05.438380 27767 consensus_queue.cc:1035] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.438597 27374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d621d98ad94cd2b9e362d64ecd2418" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" is_pre_election: true
I20260620 19:13:05.438683 27374 raft_consensus.cc:2454] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 0.
I20260620 19:13:05.438653 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d621d98ad94cd2b9e362d64ecd2418" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6" is_pre_election: true
I20260620 19:13:05.438735 27514 raft_consensus.cc:2454] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 0.
I20260620 19:13:05.438805 27587 leader_election.cc:302] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e44646adff4e2d9ba9896c5d1659b4, b033be2ec39f49b5b54c6ef9c7926bdb; no voters:
I20260620 19:13:05.438908 27735 raft_consensus.cc:2781] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:05.438939 27735 raft_consensus.cc:482] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:05.438957 27735 raft_consensus.cc:3037] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.439786 27735 raft_consensus.cc:504] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.439908 27735 leader_election.cc:288] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 election: Requested vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927)
I20260620 19:13:05.440044 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d621d98ad94cd2b9e362d64ecd2418" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6"
I20260620 19:13:05.440053 27373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d621d98ad94cd2b9e362d64ecd2418" candidate_uuid: "48e44646adff4e2d9ba9896c5d1659b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
I20260620 19:13:05.440112 27373 raft_consensus.cc:3037] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.440112 27514 raft_consensus.cc:3037] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.440822 27373 raft_consensus.cc:2454] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 1.
I20260620 19:13:05.440822 27514 raft_consensus.cc:2454] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e44646adff4e2d9ba9896c5d1659b4 in term 1.
I20260620 19:13:05.440980 27586 leader_election.cc:302] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:05.441067 27735 raft_consensus.cc:2781] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:05.441118 27735 raft_consensus.cc:686] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 LEADER]: Becoming Leader. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Running, Role: LEADER
I20260620 19:13:05.441154 27735 consensus_queue.cc:237] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.441358 27731 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: TxnStatusTablet state changed. Reason: New leader 48e44646adff4e2d9ba9896c5d1659b4. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.441418 27731 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:13:05.441596 27771 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 19:13:05.441696 27771 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 19:13:05.441669 27217 catalog_manager.cc:5495] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195). New cstate: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:05.480079 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.477895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.19.66.195:34225)
0620 19:13:05.477935 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.477937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.477968 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:13:05.478057 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.478058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.478069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.478113 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.478222 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.478689 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.479394 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.479619 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.479672 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.479848 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.479881 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.479919 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.479975 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.479993 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:13:05.480086 27762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.477846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:34225 (local address 127.0.0.1:32892)
0620 19:13:05.477919 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.477931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.477989 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.478119 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.478120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.478121 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:13:05.478173 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.478176 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.478759 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.478760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.479324 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.479326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.479679 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.479680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.479755 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.479782 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.479931 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.479936 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.479974 (+ 38us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.480017 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:13:05.481053 27514 raft_consensus.cc:1261] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Refusing update from remote peer 48e44646adff4e2d9ba9896c5d1659b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.481072 27373 raft_consensus.cc:1261] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Refusing update from remote peer 48e44646adff4e2d9ba9896c5d1659b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.481266 27735 consensus_queue.cc:1035] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.481413 27735 consensus_queue.cc:1035] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.481698 27769 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: TxnStatusTablet state changed. Reason: New leader 48e44646adff4e2d9ba9896c5d1659b4. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.481796 27769 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.481918 27767 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: TxnStatusTablet state changed. Reason: New leader 48e44646adff4e2d9ba9896c5d1659b4. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.482016 27767 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.482352 27735 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.482432 27735 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:13:05.482477 27738 mvcc.cc:205] Tried to move back new op lower bound from 7299001489328799744 to 7299001489167171584. Current Snapshot: MvccSnapshot[applied={T|T < 7299001489328799744 or (T in {7299001489328799744})}]
I20260620 19:13:05.482877 27740 mvcc.cc:205] Tried to move back new op lower bound from 7299001489328799744 to 7299001489167171584. Current Snapshot: MvccSnapshot[applied={T|T < 7299001489328799744}]
I20260620 19:13:05.482975 27769 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.483045 27769 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P b033be2ec39f49b5b54c6ef9c7926bdb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.483634 27731 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.483727 27731 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 48e44646adff4e2d9ba9896c5d1659b4: This TxnStatusTablet replica's current role is: LEADER
I20260620 19:13:05.484126 27767 tablet_replica.cc:439] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "48e44646adff4e2d9ba9896c5d1659b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } } }
I20260620 19:13:05.484210 27767 tablet_replica.cc:442] T 66d621d98ad94cd2b9e362d64ecd2418 P 372c27a542774b9bbb8db28e4548c7e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 19:13:05.495941 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.493262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.19.66.194:36927)
0620 19:13:05.493365 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.493367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.493377 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:05.493447 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.493448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.493468 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.493541 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.493691 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.494157 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.495245 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.495421 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.495463 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.495641 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.495741 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.495792 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.495858 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.495875 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260620 19:13:05.500741 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.498357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:38219 (local address 127.19.66.195:34225)
0620 19:13:05.498415 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.498417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.498550 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:13:05.498617 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.498618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.498628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.498671 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.498767 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.499254 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.500109 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.500287 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.500350 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.500502 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.500536 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.500576 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.500630 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.500647 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:13:05.500737 27785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.498290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:34225 (local address 127.19.66.194:38219)
0620 19:13:05.498469 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.498479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.498552 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.498678 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.498679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.498681 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:13:05.498721 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.498722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.499320 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.499321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.500034 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.500036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.500360 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.500361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.500422 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.500436 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.500588 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.500589 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.500629 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.500666 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260620 19:13:05.510344 27787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.507378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36927 (local address 127.19.66.195:34837)
0620 19:13:05.507562 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.507572 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.507662 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.507830 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.507830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.507832 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 19:13:05.507900 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.507902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.508535 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.508536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.509494 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.509498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.509828 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.509829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.509890 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.509907 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.510138 (+ 231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.510141 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.510208 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.510256 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260620 19:13:05.510378 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.507450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:34837 (local address 127.19.66.194:36927)
0620 19:13:05.507490 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.507492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.507634 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:13:05.507733 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.507734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.507743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.507770 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.507984 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.508452 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.509576 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.509753 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.509807 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.510057 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.510091 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.510136 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.510280 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.510307 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:05.517704 27216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.517818 27216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.518446 27216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Sending DeleteTablet for 3 replicas of tablet 9a6e4275c636418ba47833d72d059430
I20260620 19:13:05.518723 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6e4275c636418ba47833d72d059430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:38614
I20260620 19:13:05.519081 27354 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6e4275c636418ba47833d72d059430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:59236
I20260620 19:13:05.519249 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a6e4275c636418ba47833d72d059430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:32882
I20260620 19:13:05.522050 27790 tablet_replica.cc:330] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: stopping tablet replica
I20260620 19:13:05.522130 27790 raft_consensus.cc:2229] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:05.522229 27790 raft_consensus.cc:2258] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.522454 27790 ts_tablet_manager.cc:1905] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.524122 27790 ts_tablet_manager.cc:1918] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:13:05.524186 27790 log.cc:1196] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/9a6e4275c636418ba47833d72d059430
I20260620 19:13:05.524401 27790 ts_tablet_manager.cc:1939] T 9a6e4275c636418ba47833d72d059430 P 372c27a542774b9bbb8db28e4548c7e6: Deleting consensus metadata
I20260620 19:13:05.524714 27202 catalog_manager.cc:4841] TS 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927): tablet 9a6e4275c636418ba47833d72d059430 (table test-table [id=edd460eed1df43fdb822f2a0488780d7]) successfully deleted
I20260620 19:13:05.524927 27792 tablet_replica.cc:330] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: stopping tablet replica
I20260620 19:13:05.524995 27792 raft_consensus.cc:2229] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.525043 27792 raft_consensus.cc:2258] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.525285 27792 ts_tablet_manager.cc:1905] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.526407 27792 ts_tablet_manager.cc:1918] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:13:05.526458 27792 log.cc:1196] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/9a6e4275c636418ba47833d72d059430
I20260620 19:13:05.526636 27792 ts_tablet_manager.cc:1939] T 9a6e4275c636418ba47833d72d059430 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting consensus metadata
I20260620 19:13:05.526942 27204 catalog_manager.cc:4841] TS 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225): tablet 9a6e4275c636418ba47833d72d059430 (table test-table [id=edd460eed1df43fdb822f2a0488780d7]) successfully deleted
I20260620 19:13:05.528954 27791 tablet_replica.cc:330] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: stopping tablet replica
I20260620 19:13:05.529078 27791 raft_consensus.cc:2229] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.529161 27791 raft_consensus.cc:2258] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.529475 27791 ts_tablet_manager.cc:1905] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.531208 27791 ts_tablet_manager.cc:1918] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:13:05.531342 27791 log.cc:1196] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/9a6e4275c636418ba47833d72d059430
I20260620 19:13:05.531628 27791 ts_tablet_manager.cc:1939] T 9a6e4275c636418ba47833d72d059430 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting consensus metadata
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260620 19:13:05.532303 27203 catalog_manager.cc:4841] TS b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121): tablet 9a6e4275c636418ba47833d72d059430 (table test-table [id=edd460eed1df43fdb822f2a0488780d7]) successfully deleted
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 19:13:05.549847 27800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:05.542041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37886 (local address 127.19.66.254:35409)
0620 19:13:05.542136 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.542138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.542207 (+ 69us) server_negotiation.cc:407] Connection header received
0620 19:13:05.542263 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.542264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.542278 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.542359 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:05.543069 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.543838 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.544949 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.545265 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.545340 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.549774 (+ 4434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37886: BlockingRecv error: recv got EOF from 127.0.0.1:37886 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":44,"threads_started":1}
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM
I20260620 19:13:05.558632 27805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.549852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37896 (local address 127.19.66.254:35409)
0620 19:13:05.550060 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.550062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.550203 (+ 141us) server_negotiation.cc:407] Connection header received
0620 19:13:05.550224 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.550225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.550236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.550282 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:05.550858 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.551630 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.552634 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.552968 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.553023 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.555534 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:05.555536 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:05.555536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:05.555544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:05.557227 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:05.557594 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:05.557595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:05.557596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:05.557653 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:05.557828 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:05.557828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:05.557829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:05.557952 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:05.558040 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.558433 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.558525 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":48,"threads_started":1}
I20260620 19:13:05.560163 27216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:05.560338 27216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:05.561793 27354 tablet_service.cc:1467] Processing CreateTablet for tablet b98cf0d264054618b18c263351725577 (DEFAULT_TABLE table=test-table [id=12403837659b4ceeb8835f5715a360e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.561934 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98cf0d264054618b18c263351725577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.563257 27807 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Bootstrap starting.
I20260620 19:13:05.563310 27494 tablet_service.cc:1467] Processing CreateTablet for tablet b98cf0d264054618b18c263351725577 (DEFAULT_TABLE table=test-table [id=12403837659b4ceeb8835f5715a360e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.563460 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98cf0d264054618b18c263351725577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.563984 27807 tablet_bootstrap.cc:654] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.564751 27634 tablet_service.cc:1467] Processing CreateTablet for tablet b98cf0d264054618b18c263351725577 (DEFAULT_TABLE table=test-table [id=12403837659b4ceeb8835f5715a360e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.564776 27809 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Bootstrap starting.
I20260620 19:13:05.564903 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98cf0d264054618b18c263351725577. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.565421 27809 tablet_bootstrap.cc:654] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.566161 27811 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Bootstrap starting.
I20260620 19:13:05.566468 27811 tablet_bootstrap.cc:654] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.567005 27807 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: No bootstrap required, opened a new log
I20260620 19:13:05.567057 27807 ts_tablet_manager.cc:1397] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 19:13:05.567193 27807 raft_consensus.cc:348] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.567262 27807 raft_consensus.cc:374] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.567279 27807 raft_consensus.cc:729] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.567322 27807 consensus_queue.cc:260] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.567396 27807 ts_tablet_manager.cc:1428] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:05.567791 27811 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: No bootstrap required, opened a new log
I20260620 19:13:05.567836 27811 ts_tablet_manager.cc:1397] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 19:13:05.567924 27811 raft_consensus.cc:348] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.567986 27811 raft_consensus.cc:374] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.568008 27811 raft_consensus.cc:729] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.568048 27811 consensus_queue.cc:260] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.568125 27811 ts_tablet_manager.cc:1428] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.569237 27809 tablet_bootstrap.cc:492] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: No bootstrap required, opened a new log
I20260620 19:13:05.569293 27809 ts_tablet_manager.cc:1397] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 19:13:05.569417 27809 raft_consensus.cc:348] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.569486 27809 raft_consensus.cc:374] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.569504 27809 raft_consensus.cc:729] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.569537 27809 consensus_queue.cc:260] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.569625 27809 ts_tablet_manager.cc:1428] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.639846 27769 raft_consensus.cc:482] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:05.639967 27769 raft_consensus.cc:504] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.640324 27769 leader_election.cc:288] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:05.643411 27724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.640662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:36151 (local address 127.19.66.195:34225)
0620 19:13:05.640700 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.640702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.640710 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:05.640775 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.640776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.640785 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.640882 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:05.641040 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.641699 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.642689 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.643069 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.643140 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.643149 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.643299 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.643339 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:05.643421 27814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.640470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:34225 (local address 127.19.66.193:36151)
0620 19:13:05.640665 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.640673 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.640718 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.640879 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.640880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.640883 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:05.640971 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.640974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.641778 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.641779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.642589 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.642601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.643149 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.643150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.643228 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.643232 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.643277 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.643318 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":24,"thread_start_us":49,"threads_started":1}
I20260620 19:13:05.643811 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98cf0d264054618b18c263351725577" candidate_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4" is_pre_election: true
I20260620 19:13:05.643908 27654 raft_consensus.cc:2454] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b033be2ec39f49b5b54c6ef9c7926bdb in term 0.
I20260620 19:13:05.643915 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.640570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:45455 (local address 127.19.66.194:36927)
0620 19:13:05.640623 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.640624 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.640688 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:13:05.640808 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.640809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.640820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.640886 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:05.641352 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.641900 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.643064 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.643549 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.643632 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.643640 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.643816 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.643851 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:13:05.643929 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.640470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36927 (local address 127.19.66.193:45455)
0620 19:13:05.640621 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.640631 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:05.640718 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:05.640923 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:05.640924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:05.640925 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:05.641244 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.641246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.642003 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.642004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.642976 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:05.642979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.643690 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.643691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:05.643757 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.643761 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:05.643809 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:13:05.643847 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260620 19:13:05.644135 27308 leader_election.cc:302] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e44646adff4e2d9ba9896c5d1659b4, b033be2ec39f49b5b54c6ef9c7926bdb; no voters:
I20260620 19:13:05.644143 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98cf0d264054618b18c263351725577" candidate_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6" is_pre_election: true
I20260620 19:13:05.644213 27514 raft_consensus.cc:2454] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b033be2ec39f49b5b54c6ef9c7926bdb in term 0.
I20260620 19:13:05.644272 27769 raft_consensus.cc:2781] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:05.644315 27769 raft_consensus.cc:482] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:05.644335 27769 raft_consensus.cc:3037] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.645067 27769 raft_consensus.cc:504] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.645180 27769 leader_election.cc:288] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [CANDIDATE]: Term 1 election: Requested vote from peers 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:05.645324 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98cf0d264054618b18c263351725577" candidate_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4"
I20260620 19:13:05.645406 27654 raft_consensus.cc:3037] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.645383 27514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98cf0d264054618b18c263351725577" candidate_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372c27a542774b9bbb8db28e4548c7e6"
I20260620 19:13:05.645442 27514 raft_consensus.cc:3037] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.646121 27654 raft_consensus.cc:2454] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b033be2ec39f49b5b54c6ef9c7926bdb in term 1.
I20260620 19:13:05.646121 27514 raft_consensus.cc:2454] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b033be2ec39f49b5b54c6ef9c7926bdb in term 1.
I20260620 19:13:05.646291 27306 leader_election.cc:302] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, b033be2ec39f49b5b54c6ef9c7926bdb; no voters:
I20260620 19:13:05.646399 27769 raft_consensus.cc:2781] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:05.646466 27769 raft_consensus.cc:686] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 LEADER]: Becoming Leader. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Running, Role: LEADER
I20260620 19:13:05.646526 27769 consensus_queue.cc:237] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.647267 27217 catalog_manager.cc:5495] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb reported cstate change: term changed from 0 to 1, leader changed from <none> to b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193). New cstate: current_term: 1 leader_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:05.660228 27817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.658089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59240 (local address 127.19.66.193:39121)
0620 19:13:05.658237 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.658238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.658245 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:05.658278 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.658279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.658297 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.658364 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.658471 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.658946 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.659612 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.659811 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.659859 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.659967 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.660065 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.660115 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.660130 (+ 15us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.660146 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 19:13:05.661064 27514 raft_consensus.cc:1261] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Refusing update from remote peer b033be2ec39f49b5b54c6ef9c7926bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.661067 27654 raft_consensus.cc:1261] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Refusing update from remote peer b033be2ec39f49b5b54c6ef9c7926bdb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.661295 27769 consensus_queue.cc:1035] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.661414 27733 consensus_queue.cc:1035] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.664315 27216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.664415 27216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.664973 27216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Sending DeleteTablet for 3 replicas of tablet b98cf0d264054618b18c263351725577
I20260620 19:13:05.665196 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet b98cf0d264054618b18c263351725577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:38614
I20260620 19:13:05.665201 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet b98cf0d264054618b18c263351725577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:32882
I20260620 19:13:05.665297 27790 tablet_replica.cc:330] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: stopping tablet replica
I20260620 19:13:05.665352 27790 raft_consensus.cc:2229] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.665395 27790 raft_consensus.cc:2258] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.665395 27354 tablet_service.cc:1514] Processing DeleteTablet for tablet b98cf0d264054618b18c263351725577 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:59236
I20260620 19:13:05.665462 27791 tablet_replica.cc:330] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: stopping tablet replica
I20260620 19:13:05.665467 27792 tablet_replica.cc:330] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: stopping tablet replica
I20260620 19:13:05.665505 27791 raft_consensus.cc:2229] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:05.665518 27792 raft_consensus.cc:2229] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.665553 27792 raft_consensus.cc:2258] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.665576 27791 raft_consensus.cc:2258] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.665840 27790 ts_tablet_manager.cc:1905] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.665793 27216 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:05.665920 27216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:05.665917 27791 ts_tablet_manager.cc:1905] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.667164 27494 tablet_service.cc:1467] Processing CreateTablet for tablet b8593d5d9e614686a699995f84844d35 (DEFAULT_TABLE table=test-table [id=47a2f4e8ba0843e9af3d6a5ebf26c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.667308 27791 ts_tablet_manager.cc:1918] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:05.667307 27494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8593d5d9e614686a699995f84844d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.667349 27791 log.cc:1196] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/b98cf0d264054618b18c263351725577
I20260620 19:13:05.667531 27791 ts_tablet_manager.cc:1939] T b98cf0d264054618b18c263351725577 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting consensus metadata
I20260620 19:13:05.667804 27792 ts_tablet_manager.cc:1905] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.668399 27809 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Bootstrap starting.
I20260620 19:13:05.668484 27634 tablet_service.cc:1467] Processing CreateTablet for tablet b8593d5d9e614686a699995f84844d35 (DEFAULT_TABLE table=test-table [id=47a2f4e8ba0843e9af3d6a5ebf26c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.668569 27634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8593d5d9e614686a699995f84844d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.668706 27809 tablet_bootstrap.cc:654] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.669265 27792 ts_tablet_manager.cc:1918] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:05.669320 27792 log.cc:1196] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/b98cf0d264054618b18c263351725577
I20260620 19:13:05.669489 27792 ts_tablet_manager.cc:1939] T b98cf0d264054618b18c263351725577 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting consensus metadata
I20260620 19:13:05.669766 27204 catalog_manager.cc:4841] TS 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225): tablet b98cf0d264054618b18c263351725577 (table test-table [id=12403837659b4ceeb8835f5715a360e3]) successfully deleted
I20260620 19:13:05.669884 27811 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Bootstrap starting.
I20260620 19:13:05.669976 27203 catalog_manager.cc:4841] TS b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121): tablet b98cf0d264054618b18c263351725577 (table test-table [id=12403837659b4ceeb8835f5715a360e3]) successfully deleted
I20260620 19:13:05.670122 27354 tablet_service.cc:1467] Processing CreateTablet for tablet b8593d5d9e614686a699995f84844d35 (DEFAULT_TABLE table=test-table [id=47a2f4e8ba0843e9af3d6a5ebf26c9f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:05.670148 27811 tablet_bootstrap.cc:654] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.670218 27354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8593d5d9e614686a699995f84844d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:05.667311 27790 ts_tablet_manager.cc:1918] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:05.670292 27790 log.cc:1196] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/b98cf0d264054618b18c263351725577
I20260620 19:13:05.670467 27790 ts_tablet_manager.cc:1939] T b98cf0d264054618b18c263351725577 P 372c27a542774b9bbb8db28e4548c7e6: Deleting consensus metadata
I20260620 19:13:05.670702 27202 catalog_manager.cc:4841] TS 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927): tablet b98cf0d264054618b18c263351725577 (table test-table [id=12403837659b4ceeb8835f5715a360e3]) successfully deleted
I20260620 19:13:05.671017 27809 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: No bootstrap required, opened a new log
I20260620 19:13:05.671017 27811 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: No bootstrap required, opened a new log
I20260620 19:13:05.671056 27811 ts_tablet_manager.cc:1397] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:13:05.671056 27809 ts_tablet_manager.cc:1397] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:13:05.671152 27811 raft_consensus.cc:348] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.671159 27809 raft_consensus.cc:348] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.671209 27811 raft_consensus.cc:374] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.671212 27809 raft_consensus.cc:374] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.671231 27809 raft_consensus.cc:729] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.671228 27811 raft_consensus.cc:729] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e44646adff4e2d9ba9896c5d1659b4, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.671271 27809 consensus_queue.cc:260] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.671273 27811 consensus_queue.cc:260] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.671350 27809 ts_tablet_manager.cc:1428] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.671350 27811 ts_tablet_manager.cc:1428] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.671428 27807 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: Bootstrap starting.
I20260620 19:13:05.671814 27807 tablet_bootstrap.cc:654] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:05.672685 27807 tablet_bootstrap.cc:492] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: No bootstrap required, opened a new log
I20260620 19:13:05.672734 27807 ts_tablet_manager.cc:1397] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:05.672907 27807 raft_consensus.cc:348] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.672973 27807 raft_consensus.cc:374] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:05.672991 27807 raft_consensus.cc:729] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b033be2ec39f49b5b54c6ef9c7926bdb, State: Initialized, Role: FOLLOWER
I20260620 19:13:05.673025 27807 consensus_queue.cc:260] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.673110 27807 ts_tablet_manager.cc:1428] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:05.724773 27767 raft_consensus.cc:482] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:05.724887 27767 raft_consensus.cc:504] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.725025 27767 leader_election.cc:288] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:05.725222 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8593d5d9e614686a699995f84844d35" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4" is_pre_election: true
I20260620 19:13:05.725222 27373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8593d5d9e614686a699995f84844d35" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" is_pre_election: true
I20260620 19:13:05.725315 27373 raft_consensus.cc:2454] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 0.
I20260620 19:13:05.725315 27654 raft_consensus.cc:2454] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 0.
I20260620 19:13:05.725466 27448 leader_election.cc:302] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, 48e44646adff4e2d9ba9896c5d1659b4; no voters:
I20260620 19:13:05.725553 27767 raft_consensus.cc:2781] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:05.725585 27767 raft_consensus.cc:482] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:05.725603 27767 raft_consensus.cc:3037] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.726281 27767 raft_consensus.cc:504] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.726397 27767 leader_election.cc:288] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 election: Requested vote from peers b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121), 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225)
I20260620 19:13:05.726572 27373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8593d5d9e614686a699995f84844d35" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb"
I20260620 19:13:05.726634 27373 raft_consensus.cc:3037] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.726584 27654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8593d5d9e614686a699995f84844d35" candidate_uuid: "372c27a542774b9bbb8db28e4548c7e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48e44646adff4e2d9ba9896c5d1659b4"
I20260620 19:13:05.726673 27654 raft_consensus.cc:3037] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:05.727202 27373 raft_consensus.cc:2454] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 1.
I20260620 19:13:05.727357 27447 leader_election.cc:302] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372c27a542774b9bbb8db28e4548c7e6, b033be2ec39f49b5b54c6ef9c7926bdb; no voters:
I20260620 19:13:05.727427 27767 raft_consensus.cc:2781] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:05.727463 27767 raft_consensus.cc:686] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 LEADER]: Becoming Leader. State: Replica: 372c27a542774b9bbb8db28e4548c7e6, State: Running, Role: LEADER
I20260620 19:13:05.727483 27654 raft_consensus.cc:2454] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372c27a542774b9bbb8db28e4548c7e6 in term 1.
I20260620 19:13:05.727509 27767 consensus_queue.cc:237] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } }
I20260620 19:13:05.728062 27216 catalog_manager.cc:5495] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "372c27a542774b9bbb8db28e4548c7e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372c27a542774b9bbb8db28e4548c7e6" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 } health_report { overall_health: UNKNOWN } } }
I20260620 19:13:05.748502 27782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.746467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.19.66.194:36927)
0620 19:13:05.746508 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:05.746510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:05.746517 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:05.746538 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:05.746539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:05.746548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:05.746600 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:05.746690 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.747217 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.747901 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:05.748099 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:05.748152 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:05.748274 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:05.748308 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:05.748349 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:05.748405 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:13:05.748425 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:13:05.750515 27373 raft_consensus.cc:1261] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Refusing update from remote peer 372c27a542774b9bbb8db28e4548c7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.750658 27654 raft_consensus.cc:1261] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Refusing update from remote peer 372c27a542774b9bbb8db28e4548c7e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:05.750730 27755 consensus_queue.cc:1035] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b033be2ec39f49b5b54c6ef9c7926bdb" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.750829 27767 consensus_queue.cc:1035] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48e44646adff4e2d9ba9896c5d1659b4" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 34225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:05.760684 27216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.760779 27216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:05.761327 27216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b77c89e327d4f6fb0532090c539a0d5: Sending DeleteTablet for 3 replicas of tablet b8593d5d9e614686a699995f84844d35
I20260620 19:13:05.761560 27494 tablet_service.cc:1514] Processing DeleteTablet for tablet b8593d5d9e614686a699995f84844d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:38614
I20260620 19:13:05.761626 27354 tablet_service.cc:1514] Processing DeleteTablet for tablet b8593d5d9e614686a699995f84844d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:59236
I20260620 19:13:05.761651 27634 tablet_service.cc:1514] Processing DeleteTablet for tablet b8593d5d9e614686a699995f84844d35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:05 UTC) from {username='oryx', principal='oryx/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:32882
I20260620 19:13:05.761705 27790 tablet_replica.cc:330] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: stopping tablet replica
I20260620 19:13:05.761719 27792 tablet_replica.cc:330] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: stopping tablet replica
I20260620 19:13:05.761709 27791 tablet_replica.cc:330] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: stopping tablet replica
I20260620 19:13:05.761756 27790 raft_consensus.cc:2229] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:05.761759 27792 raft_consensus.cc:2229] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.761771 27791 raft_consensus.cc:2229] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:05.761791 27792 raft_consensus.cc:2258] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.761804 27790 raft_consensus.cc:2258] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.761809 27791 raft_consensus.cc:2258] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:05.762069 27790 ts_tablet_manager.cc:1905] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.762337 27792 ts_tablet_manager.cc:1905] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:05.763554 27790 ts_tablet_manager.cc:1918] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:13:05.763614 27790 log.cc:1196] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/b8593d5d9e614686a699995f84844d35
I20260620 19:13:05.763779 27792 ts_tablet_manager.cc:1918] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 19:13:05.763810 27790 ts_tablet_manager.cc:1939] T b8593d5d9e614686a699995f84844d35 P 372c27a542774b9bbb8db28e4548c7e6: Deleting consensus metadata
I20260620 19:13:05.763834 27792 log.cc:1196] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/b8593d5d9e614686a699995f84844d35
I20260620 19:13:05.764021 27792 ts_tablet_manager.cc:1939] T b8593d5d9e614686a699995f84844d35 P 48e44646adff4e2d9ba9896c5d1659b4: Deleting consensus metadata
I20260620 19:13:05.764127 27202 catalog_manager.cc:4841] TS 372c27a542774b9bbb8db28e4548c7e6 (127.19.66.194:36927): tablet b8593d5d9e614686a699995f84844d35 (table test-table [id=47a2f4e8ba0843e9af3d6a5ebf26c9f4]) successfully deleted
I20260620 19:13:05.764377 27204 catalog_manager.cc:4841] TS 48e44646adff4e2d9ba9896c5d1659b4 (127.19.66.195:34225): tablet b8593d5d9e614686a699995f84844d35 (table test-table [id=47a2f4e8ba0843e9af3d6a5ebf26c9f4]) successfully deleted
I20260620 19:13:05.764569 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27289
I20260620 19:13:05.764669 27791 ts_tablet_manager.cc:1905] T b8593d5d9e614686a699995f84844d35 P b033be2ec39f49b5b54c6ef9c7926bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:13:05.769925 27203 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:39121 (error 108)
I20260620 19:13:05.770076 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27429
I20260620 19:13:05.771243 27829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:05.771001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:39121 (local address 127.0.0.1:59254)
0620 19:13:05.771148 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:05.771201 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:39121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
W20260620 19:13:05.771757 27203 catalog_manager.cc:4584] TS b033be2ec39f49b5b54c6ef9c7926bdb (127.19.66.193:39121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8593d5d9e614686a699995f84844d35: Network error: Client connection negotiation failed: client connection to 127.19.66.193:39121: connect: Connection refused (error 111)
I20260620 19:13:05.775344 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27569
I20260620 19:13:05.780094 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27185
2026-06-20T19:13:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2864 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27832](info): set up 2 sockets
Jun 20 19:13:05 dist-test-slave-nq8r krb5kdc[27832](info): commencing operation
krb5kdc: starting...
W20260620 19:13:06.721043 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:13:06 dist-test-slave-nq8r krb5kdc[27832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:13:06.825856 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35227
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:06.904151 27845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:06.904347 27845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:06.904376 27845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:06.905539 27845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:06.905578 27845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:06.905598 27845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:06.905614 27845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:06.905629 27845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:06.907084 27845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:06.907316 27845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:06.907505 27845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:06.911939 27853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:06.911952 27850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:06.912019 27845 server_base.cc:1034] running on GCE node
W20260620 19:13:06.911933 27851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:06.912403 27845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:06.912684 27845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:06.913828 27845 hybrid_clock.cc:648] HybridClock initialized: now 1781982786913804 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T19:13:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1143 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:06 dist-test-slave-nq8r krb5kdc[27858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:06 dist-test-slave-nq8r krb5kdc[27858](info): set up 2 sockets
Jun 20 19:13:06 dist-test-slave-nq8r krb5kdc[27858](info): commencing operation
krb5kdc: starting...
W20260620 19:13:07.873109 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:13:07 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:13:07.977411 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33277
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:08.054729 27871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.054925 27871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.054945 27871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.056078 27871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:08.056118 27871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.056130 27871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.056145 27871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:08.056169 27871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:08.057695 27871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.057921 27871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.058120 27871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.062544 27879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.062610 27877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.062582 27876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.062724 27871 server_base.cc:1034] running on GCE node
I20260620 19:13:08.062942 27871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.063236 27871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.064365 27871 hybrid_clock.cc:648] HybridClock initialized: now 1781982788064348 us; error 34 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.066696 27871 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:13:08.067130 27871 webserver.cc:460] Webserver started at http://127.19.66.254:39381/ using document root <none> and password file <none>
I20260620 19:13:08.067356 27871 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.067404 27871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.067520 27871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:08.068533 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f400e0d0133f487d83cc9eb26cdc3708"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "8ba00839ac80f3349a4bc55c911e7ec7"
server_key_iv: "9cfe168062efa3245371c8c65e6d4e0c"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.068842 27871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f400e0d0133f487d83cc9eb26cdc3708"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "8ba00839ac80f3349a4bc55c911e7ec7"
server_key_iv: "9cfe168062efa3245371c8c65e6d4e0c"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.070070 27871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:08.070713 27886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.070863 27871 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:08.070941 27871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f400e0d0133f487d83cc9eb26cdc3708"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "8ba00839ac80f3349a4bc55c911e7ec7"
server_key_iv: "9cfe168062efa3245371c8c65e6d4e0c"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.071000 27871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.078339 27871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.080169 27871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.080302 27871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.084084 27871 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:33277
I20260620 19:13:08.084126 27938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:33277 every 8 connection(s)
I20260620 19:13:08.084491 27871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:08.085350 27939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:08.087523 27939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Bootstrap starting.
I20260620 19:13:08.088066 27939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:08.088292 27939 log.cc:824] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:08.088922 27939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: No bootstrap required, opened a new log
I20260620 19:13:08.090442 27939 raft_consensus.cc:348] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.090580 27939 raft_consensus.cc:374] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:08.090608 27939 raft_consensus.cc:729] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f400e0d0133f487d83cc9eb26cdc3708, State: Initialized, Role: FOLLOWER
I20260620 19:13:08.090713 27939 consensus_queue.cc:260] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.090761 27939 raft_consensus.cc:388] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:08.090785 27939 raft_consensus.cc:482] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:08.090814 27939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:08.091305 27939 raft_consensus.cc:504] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.091372 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27871
I20260620 19:13:08.091413 27939 leader_election.cc:302] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f400e0d0133f487d83cc9eb26cdc3708; no voters:
I20260620 19:13:08.091459 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:13:08.091570 27939 leader_election.cc:288] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:08.091595 27942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:08.091655 19723 external_mini_cluster.cc:1428] Setting key a18a221386aad91eb061ef76bb3454ed
I20260620 19:13:08.091775 27942 raft_consensus.cc:686] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 LEADER]: Becoming Leader. State: Replica: f400e0d0133f487d83cc9eb26cdc3708, State: Running, Role: LEADER
I20260620 19:13:08.091809 27939 sys_catalog.cc:564] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:08.092504 27942 consensus_queue.cc:237] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.093171 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f400e0d0133f487d83cc9eb26cdc3708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } } }
I20260620 19:13:08.093346 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:08.093029 27943 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f400e0d0133f487d83cc9eb26cdc3708. Latest consensus state: current_term: 1 leader_uuid: "f400e0d0133f487d83cc9eb26cdc3708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } } }
I20260620 19:13:08.093609 27943 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: This master's current role is: LEADER
W20260620 19:13:08.093896 27958 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:13:08.093937 27958 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:13:08.094013 27960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:08.094182 27960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:08.095420 27960 catalog_manager.cc:1347] Generated new cluster ID: c0c57ee2f12848d5822c333ed200d502
I20260620 19:13:08.095466 27960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:08.103523 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.092366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.19.66.254:33277)
0620 19:13:08.092879 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.092888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.092922 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:13:08.093047 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.093054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.093109 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.094058 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.094150 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.095518 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.096559 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.097202 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.097320 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.100005 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:08.100009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:08.100009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:08.100022 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:08.101927 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.102310 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.102312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.102312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.102422 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.102622 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.102623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.102623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.102908 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:08.103042 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:08.103160 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:13:08.103287 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":81,"threads_started":1}
I20260620 19:13:08.107570 27960 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:08.108058 27960 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:08.111464 27960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Generated new TSK 0
I20260620 19:13:08.111614 27960 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:13:08.139298 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:08.216576 27965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.216773 27965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.216804 27965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.217937 27965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.217975 27965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.218010 27965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:08.219381 27965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.219604 27965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.219808 27965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.224306 27973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.224380 27965 server_base.cc:1034] running on GCE node
W20260620 19:13:08.224288 27971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.224295 27970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.224673 27965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.224921 27965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.226058 27965 hybrid_clock.cc:648] HybridClock initialized: now 1781982788226049 us; error 26 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.228243 27965 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:13:08.228590 27965 webserver.cc:460] Webserver started at http://127.19.66.193:46109/ using document root <none> and password file <none>
I20260620 19:13:08.228758 27965 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.228796 27965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.228916 27965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:08.229872 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "0ffe2a8d60384860a7ca1ca3d16ac78a"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "5d187846a4d5ee758a837e3f8a435d5b"
server_key_iv: "e219e28f93092042da2231c0c20f0d19"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.230156 27965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "0ffe2a8d60384860a7ca1ca3d16ac78a"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "5d187846a4d5ee758a837e3f8a435d5b"
server_key_iv: "e219e28f93092042da2231c0c20f0d19"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.231238 27965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:13:08.231827 27980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.231959 27965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:08.232034 27965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "0ffe2a8d60384860a7ca1ca3d16ac78a"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "5d187846a4d5ee758a837e3f8a435d5b"
server_key_iv: "e219e28f93092042da2231c0c20f0d19"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.232085 27965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.243337 27965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.245397 27965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.245498 27965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.245666 27965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:08.245900 27965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:08.245932 27965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.245951 27965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:08.245967 27965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.251060 27965 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:40771
I20260620 19:13:08.251106 28093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:40771 every 8 connection(s)
I20260620 19:13:08.251381 27965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:08.253788 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 27965
I20260620 19:13:08.253877 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:08.254043 19723 external_mini_cluster.cc:1428] Setting key 7732526c8effc45fa0a95415a0697771
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:08.260501 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.252031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:56309 (local address 127.19.66.254:33277)
0620 19:13:08.252107 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.252110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.252285 (+ 175us) server_negotiation.cc:407] Connection header received
0620 19:13:08.252725 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.252726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.252735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.252782 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.253646 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.254275 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.255573 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.255768 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.255835 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.258081 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:08.258083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:08.258083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:08.258090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:08.259298 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.259659 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.259661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.259661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.259709 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.259917 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.259918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.259918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.260021 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:08.260077 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:08.260332 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 19:13:08.260414 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 19:13:08.260660 28096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.252002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:56309)
0620 19:13:08.252172 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.252206 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.252653 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.252836 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.252839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.253375 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.253556 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.253560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.254359 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.254360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.255484 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.255486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.255849 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.255850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.255997 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.256454 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.256461 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.257982 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:08.259376 (+ 1394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.259384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.259386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.259588 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.259762 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.259763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.259763 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.259863 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.260074 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:08.260080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:08.260218 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 19:13:08.260304 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:13:08.260452 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260620 19:13:08.261368 28094 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33277
I20260620 19:13:08.261454 28094 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:08.261619 28094 heartbeater.cc:511] Master 127.19.66.254:33277 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
I20260620 19:13:08.262102 27903 ts_manager.cc:194] Registered new tserver with Master: 0ffe2a8d60384860a7ca1ca3d16ac78a (127.19.66.193:40771)
I20260620 19:13:08.262686 27903 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:56309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:13:08.275106 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:08.352257 28099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.352463 28099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.352494 28099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.353684 28099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.353724 28099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.353760 28099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:08.355151 28099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.355386 28099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.355592 28099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.360167 28105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.360167 28107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.360334 28104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.360395 28099 server_base.cc:1034] running on GCE node
I20260620 19:13:08.360605 28099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.360823 28099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.362008 28099 hybrid_clock.cc:648] HybridClock initialized: now 1781982788361994 us; error 26 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.364256 28099 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:13:08.364661 28099 webserver.cc:460] Webserver started at http://127.19.66.194:35859/ using document root <none> and password file <none>
I20260620 19:13:08.364840 28099 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.364914 28099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.365012 28099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:08.365964 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "99bff2eceb584e26b7815307df86330f"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "7d17dae424105ea2ba2fb253fbca8d57"
server_key_iv: "c1f4e18b028f1cd097522ea276c0e821"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.366248 28099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "99bff2eceb584e26b7815307df86330f"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "7d17dae424105ea2ba2fb253fbca8d57"
server_key_iv: "c1f4e18b028f1cd097522ea276c0e821"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.367329 28099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:13:08.367908 28114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.368067 28099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:13:08.368142 28099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "99bff2eceb584e26b7815307df86330f"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "7d17dae424105ea2ba2fb253fbca8d57"
server_key_iv: "c1f4e18b028f1cd097522ea276c0e821"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.368199 28099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.382412 28099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.384004 28099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.384122 28099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.384305 28099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:08.384573 28099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:08.384604 28099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.384631 28099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:08.384655 28099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.389951 28099 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38987
I20260620 19:13:08.390003 28227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38987 every 8 connection(s)
I20260620 19:13:08.390280 28099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:08.398558 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.390927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:49093 (local address 127.19.66.254:33277)
0620 19:13:08.390970 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.390971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.391163 (+ 192us) server_negotiation.cc:407] Connection header received
0620 19:13:08.391699 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.391700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.391712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.391778 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.392684 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.393284 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.394245 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.394446 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.394507 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.396320 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:08.396322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:08.396322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:08.396329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:08.397403 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.397736 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.397737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.397737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.397790 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.397987 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.397987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.397988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.398091 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:08.398146 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:08.398384 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 19:13:08.398462 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:08.398711 28230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.390893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:49093)
0620 19:13:08.391038 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.391078 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.391611 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.391846 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.391850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.392412 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.392596 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.392600 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.393367 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.393368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.394164 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.394166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.394511 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.394512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.394640 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.395040 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.395047 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.396238 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:08.397469 (+ 1231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.397475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.397477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.397675 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.397839 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.397840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.397840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.397935 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.398142 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:08.398147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:08.398272 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 19:13:08.398355 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 19:13:08.398493 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 19:13:08.398891 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28099
I20260620 19:13:08.398995 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:08.399080 28228 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33277
I20260620 19:13:08.399168 28228 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:08.399184 19723 external_mini_cluster.cc:1428] Setting key 573df0ce0e3a748890059879d1e0a77d
I20260620 19:13:08.399325 28228 heartbeater.cc:511] Master 127.19.66.254:33277 requested a full tablet report, sending...
I20260620 19:13:08.399686 27902 ts_manager.cc:194] Registered new tserver with Master: 99bff2eceb584e26b7815307df86330f (127.19.66.194:38987)
I20260620 19:13:08.400033 27902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:49093
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:13:08.419457 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:08.496621 28233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.496830 28233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.496869 28233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.498011 28233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.498051 28233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.498076 28233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:08.499473 28233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.499701 28233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.499903 28233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:13:08.504545 28233 server_base.cc:1034] running on GCE node
W20260620 19:13:08.504519 28238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.504539 28241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.504534 28239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.504936 28233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.505167 28233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.506294 28233 hybrid_clock.cc:648] HybridClock initialized: now 1781982788506284 us; error 27 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.508610 28233 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:13:08.509068 28233 webserver.cc:460] Webserver started at http://127.19.66.195:45681/ using document root <none> and password file <none>
I20260620 19:13:08.509279 28233 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.509321 28233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.509433 28233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:08.510444 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "14a18a72a17843da834e0042173e94a0"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "d120cb97a52d9a47b295bd6fd9f22bf4"
server_key_iv: "8131d2b94a0467e07a24d05237417ecc"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.510769 28233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "14a18a72a17843da834e0042173e94a0"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "d120cb97a52d9a47b295bd6fd9f22bf4"
server_key_iv: "8131d2b94a0467e07a24d05237417ecc"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.511943 28233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:08.512542 28248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.512679 28233 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:08.512758 28233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "14a18a72a17843da834e0042173e94a0"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "d120cb97a52d9a47b295bd6fd9f22bf4"
server_key_iv: "8131d2b94a0467e07a24d05237417ecc"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.512817 28233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.525942 28233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.528079 28233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.528211 28233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.528414 28233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:08.528700 28233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:08.528733 28233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.528762 28233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:08.528787 28233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.534154 28233 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:40449
I20260620 19:13:08.534202 28361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:40449 every 8 connection(s)
I20260620 19:13:08.534490 28233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:08.542714 27950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.535144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:35501 (local address 127.19.66.254:33277)
0620 19:13:08.535204 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.535207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.535387 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:13:08.535907 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.535907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.535916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.535974 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.536890 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.537390 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.538344 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.538519 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.538565 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.540535 (+ 1970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:08.540536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:08.540537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:08.540544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:08.541683 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.542009 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.542010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.542010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.542061 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:08.542238 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:08.542238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:08.542238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:08.542339 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:08.542387 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:08.542587 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 19:13:08.542648 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:13:08.542865 28364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.535121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:35501)
0620 19:13:08.535267 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.535308 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.535825 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.536030 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.536034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.536595 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.536787 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.536792 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.537464 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.537465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.538280 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.538282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.538573 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.538573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.538667 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.539153 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.539162 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.540446 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:08.541745 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.541751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.541753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.541955 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.542104 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:08.542105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:08.542105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:08.542195 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:08.542381 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:08.542384 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:08.542492 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 19:13:08.542569 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:13:08.542681 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260620 19:13:08.543289 28362 heartbeater.cc:347] Connected to a master server at 127.19.66.254:33277
I20260620 19:13:08.543377 28362 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:08.543535 28362 heartbeater.cc:511] Master 127.19.66.254:33277 requested a full tablet report, sending...
I20260620 19:13:08.543601 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28233
I20260620 19:13:08.543705 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:08.543877 19723 external_mini_cluster.cc:1428] Setting key fb0ae1bd8f07b06d98bf9745f3d801de
I20260620 19:13:08.543888 27902 ts_manager.cc:194] Registered new tserver with Master: 14a18a72a17843da834e0042173e94a0 (127.19.66.195:40449)
I20260620 19:13:08.544324 27902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:35501
I20260620 19:13:08.544644 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
Entry for principal oryx/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab.
WARNING: no policy specified for oryx/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
Entry for principal oryx/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab.
WARNING: no policy specified for oryx/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
Entry for principal oryx/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab.
I20260620 19:13:08.604071 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27965
I20260620 19:13:08.607910 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28099
I20260620 19:13:08.611726 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28233
I20260620 19:13:08.615216 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 27871
I20260620 19:13:08.619537 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33277
--webserver_interface=127.19.66.254
--webserver_port=39381
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:08.697525 28371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.697723 28371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.697752 28371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.698868 28371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:08.698906 28371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.698925 28371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.698940 28371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:08.698956 28371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:08.700394 28371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.700627 28371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.700814 28371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.705232 28376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.705242 28379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.705404 28371 server_base.cc:1034] running on GCE node
W20260620 19:13:08.705242 28377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.705673 28371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.705936 28371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.707094 28371 hybrid_clock.cc:648] HybridClock initialized: now 1781982788707078 us; error 27 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.709443 28371 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:13:08.709846 28371 webserver.cc:460] Webserver started at http://127.19.66.254:39381/ using document root <none> and password file <none>
I20260620 19:13:08.710048 28371 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.710117 28371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.711383 28371 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:08.711926 28386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.712082 28371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.712152 28371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f400e0d0133f487d83cc9eb26cdc3708"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "8ba00839ac80f3349a4bc55c911e7ec7"
server_key_iv: "9cfe168062efa3245371c8c65e6d4e0c"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.712471 28371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.721966 28371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.723706 28371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.723825 28371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.727445 28371 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:33277
I20260620 19:13:08.727476 28438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:33277 every 8 connection(s)
I20260620 19:13:08.727803 28371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:08.729174 28439 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:13:08.729784 28439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Bootstrap starting.
I20260620 19:13:08.733539 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28371
I20260620 19:13:08.733789 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:40771
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:33277
--webserver_port=46109
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:13:08.737789 28439 log.cc:824] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:08.739405 28439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:13:08.739656 28439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Bootstrap complete.
I20260620 19:13:08.741712 28439 raft_consensus.cc:348] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.741923 28439 raft_consensus.cc:729] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f400e0d0133f487d83cc9eb26cdc3708, State: Initialized, Role: FOLLOWER
I20260620 19:13:08.742064 28439 consensus_queue.cc:260] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.742129 28439 raft_consensus.cc:388] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:08.742168 28439 raft_consensus.cc:482] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:08.742215 28439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:13:08.743167 28439 raft_consensus.cc:504] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.743254 28439 leader_election.cc:302] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f400e0d0133f487d83cc9eb26cdc3708; no voters:
I20260620 19:13:08.743403 28439 leader_election.cc:288] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:13:08.743455 28443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:13:08.743613 28443 raft_consensus.cc:686] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [term 2 LEADER]: Becoming Leader. State: Replica: f400e0d0133f487d83cc9eb26cdc3708, State: Running, Role: LEADER
I20260620 19:13:08.743680 28439 sys_catalog.cc:564] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:08.743788 28443 consensus_queue.cc:237] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } }
I20260620 19:13:08.744155 28445 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f400e0d0133f487d83cc9eb26cdc3708. Latest consensus state: current_term: 2 leader_uuid: "f400e0d0133f487d83cc9eb26cdc3708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } } }
I20260620 19:13:08.744167 28444 sys_catalog.cc:455] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f400e0d0133f487d83cc9eb26cdc3708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f400e0d0133f487d83cc9eb26cdc3708" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 33277 } } }
I20260620 19:13:08.744272 28444 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:08.744233 28445 sys_catalog.cc:458] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:08.744835 28452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:08.745309 28452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:08.745426 28452 catalog_manager.cc:1259] Loaded cluster ID: c0c57ee2f12848d5822c333ed200d502
I20260620 19:13:08.745462 28452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:08.746222 28452 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:08.746367 28452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f400e0d0133f487d83cc9eb26cdc3708: Loaded TSK: 0
I20260620 19:13:08.746548 28452 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:13:08.831292 28441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.831496 28441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.831528 28441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.832690 28441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.832732 28441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.832767 28441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:08.834204 28441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.193.keytab
--principal=oryx/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:40771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=46109
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.834435 28441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.834640 28441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.839347 28468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.839370 28466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.839354 28465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.839797 28441 server_base.cc:1034] running on GCE node
I20260620 19:13:08.840021 28441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.840286 28441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.841457 28441 hybrid_clock.cc:648] HybridClock initialized: now 1781982788841437 us; error 34 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.844329 28441 init.cc:377] Logged in from keytab as oryx/127.19.66.193@KRBTEST.COM (short username oryx)
I20260620 19:13:08.844728 28441 webserver.cc:460] Webserver started at http://127.19.66.193:46109/ using document root <none> and password file <none>
I20260620 19:13:08.844993 28441 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.845079 28441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.846421 28441 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:08.847018 28475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.847173 28441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:08.847249 28441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "0ffe2a8d60384860a7ca1ca3d16ac78a"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "5d187846a4d5ee758a837e3f8a435d5b"
server_key_iv: "e219e28f93092042da2231c0c20f0d19"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.847569 28441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.867390 28441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.869444 28441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.869571 28441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.869776 28441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:08.870041 28441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:08.870072 28441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.870100 28441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:08.870124 28441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.875372 28441 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:40771
I20260620 19:13:08.875469 28588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:40771 every 8 connection(s)
I20260620 19:13:08.875749 28441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:08.879140 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28441
I20260620 19:13:08.879407 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:38987
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:33277
--webserver_port=35859
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:08.885098 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.876390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:57361)
0620 19:13:08.876566 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.876607 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.877145 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.877579 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.877582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.878110 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.878302 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.878307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.879908 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.879909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.880991 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.880995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.881560 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.881562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.881690 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.882174 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.882184 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.884884 (+ 2700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
W20260620 19:13:08.885207 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:08.876473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:57361 (local address 127.19.66.254:33277)
0620 19:13:08.876656 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.876664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.876698 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:13:08.877346 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.877353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.877405 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.877552 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.878409 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.879807 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.881068 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.881484 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.881578 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.885035 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:57361: BlockingRecv error: recv got EOF from 127.19.66.193:57361 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
W20260620 19:13:08.885390 28589 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:33277 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.66.254:33277: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:08.895507 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.885601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:42611)
0620 19:13:08.891419 (+ 5818us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.891431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.891493 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.891621 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.891622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.891754 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.891804 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.891806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.892417 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.892418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.892960 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.892963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.893283 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.893283 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.893344 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.893783 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.893787 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.895439 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5801}
W20260620 19:13:08.895576 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:08.885649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:42611 (local address 127.19.66.254:33277)
0620 19:13:08.885683 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.885684 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.891499 (+ 5815us) server_negotiation.cc:407] Connection header received
0620 19:13:08.891551 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.891552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.891571 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.891618 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.891869 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.892346 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.893026 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.893225 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.893279 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.895526 (+ 2247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:42611: BlockingRecv error: recv got EOF from 127.19.66.193:42611 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:08.899840 28591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:08.895819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:60481)
0620 19:13:08.895878 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:08.895887 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:08.895918 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:08.896036 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:08.896037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:08.896174 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:08.896208 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.896210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.896772 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.896772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.897591 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:08.897593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.897883 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.897884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:08.897935 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.898268 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:08.898273 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:08.899777 (+ 1504us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 19:13:08.899900 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:08.895868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:60481 (local address 127.19.66.254:33277)
0620 19:13:08.895900 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:08.895901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:08.895909 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:08.895970 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:08.895970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:08.895977 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:08.896020 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:08.896266 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.896714 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.897663 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:08.897831 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:08.897872 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:08.899853 (+ 1981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:60481: BlockingRecv error: recv got EOF from 127.19.66.193:60481 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:13:08.899964 28589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:13:08.960516 28593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:08.960712 28593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:08.960742 28593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:08.961881 28593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:08.961920 28593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:08.961954 28593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:08.963320 28593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.194.keytab
--principal=oryx/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:38987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=35859
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:08.963541 28593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:08.963740 28593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:08.968235 28601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.968241 28598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:08.968320 28599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:08.968335 28593 server_base.cc:1034] running on GCE node
I20260620 19:13:08.968726 28593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:08.968999 28593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:08.970151 28593 hybrid_clock.cc:648] HybridClock initialized: now 1781982788970134 us; error 29 us; skew 500 ppm
Jun 20 19:13:08 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982788, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:08.972492 28593 init.cc:377] Logged in from keytab as oryx/127.19.66.194@KRBTEST.COM (short username oryx)
I20260620 19:13:08.972945 28593 webserver.cc:460] Webserver started at http://127.19.66.194:35859/ using document root <none> and password file <none>
I20260620 19:13:08.973208 28593 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:08.973284 28593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:08.974673 28593 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:08.980541 28608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.981822 28593 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260620 19:13:08.981954 28593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "99bff2eceb584e26b7815307df86330f"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "7d17dae424105ea2ba2fb253fbca8d57"
server_key_iv: "c1f4e18b028f1cd097522ea276c0e821"
server_key_version: "encryptionkey@0"
I20260620 19:13:08.982405 28593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:08.997069 28593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:08.998798 28593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:08.998906 28593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:08.999082 28593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:08.999377 28593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:08.999414 28593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:08.999444 28593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:08.999470 28593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:09.005100 28593 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38987
I20260620 19:13:09.005136 28721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38987 every 8 connection(s)
I20260620 19:13:09.005482 28593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.012351 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.006202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:40515)
0620 19:13:09.006330 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.006371 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.006831 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.007060 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.007063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.007629 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.007825 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.007830 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.008515 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.008516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.009336 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.009338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.009656 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.009657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.009753 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.010234 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.010243 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.012201 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":62,"threads_started":1}
W20260620 19:13:09.012346 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.006203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:40515 (local address 127.19.66.254:33277)
0620 19:13:09.006262 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.006264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.006463 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:13:09.006908 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.006909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.006924 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.006975 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.007926 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.008444 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.009401 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.009597 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.009651 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.012296 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:40515: BlockingRecv error: recv got EOF from 127.19.66.194:40515 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 19:13:09.012594 28722 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:33277 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.66.254:33277: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
I20260620 19:13:09.014778 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28593
I20260620 19:13:09.015075 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:40449
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:33277
--webserver_port=45681
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:42127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.016497 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.012778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:34445)
0620 19:13:09.012817 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.012827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.012887 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.012999 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.013000 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.013100 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.013142 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.013143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.013701 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.013701 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.014180 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.014181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.014463 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.014464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.014504 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.014897 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.014902 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.016426 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:13:09.016572 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.012827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:34445 (local address 127.19.66.254:33277)
0620 19:13:09.012883 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.012885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.012893 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:09.012935 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.012936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.012943 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.012994 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.013201 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.013648 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.014241 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.014413 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.014463 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.016524 (+ 2061us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:34445: BlockingRecv error: recv got EOF from 127.19.66.194:34445 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.020419 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.016729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:57991)
0620 19:13:09.016868 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.016876 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.016916 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.017019 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.017019 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.017131 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.017159 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.017160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.017720 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.017721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.018194 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.018195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.018490 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.018490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.018536 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.018892 (+ 356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.018896 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.020376 (+ 1480us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
W20260620 19:13:09.020517 28722 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:13:09.020535 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.016777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:57991 (local address 127.19.66.254:33277)
0620 19:13:09.016807 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.016809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.016905 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:13:09.016966 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.016967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.016975 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.017018 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.017218 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.017659 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.018257 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.018439 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.018487 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.020478 (+ 1991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:57991: BlockingRecv error: recv got EOF from 127.19.66.194:57991 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:13:09.092947 28725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:09.093152 28725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:09.093175 28725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:09.094301 28725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:09.094336 28725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:09.094362 28725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:09.095750 28725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:42127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/krb5kdc/oryx_127.19.66.195.keytab
--principal=oryx/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:40449
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=45681
--tserver_master_addrs=127.19.66.254:33277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:09.095943 28725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:09.096127 28725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:09.100824 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:09.100826 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:09.100824 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:09.100878 28725 server_base.cc:1034] running on GCE node
I20260620 19:13:09.101261 28725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:09.101498 28725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:09.102666 28725 hybrid_clock.cc:648] HybridClock initialized: now 1781982789102649 us; error 28 us; skew 500 ppm
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982789, etypes {rep=17 tkt=17 ses=17}, oryx/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:09.105062 28725 init.cc:377] Logged in from keytab as oryx/127.19.66.195@KRBTEST.COM (short username oryx)
I20260620 19:13:09.105485 28725 webserver.cc:460] Webserver started at http://127.19.66.195:45681/ using document root <none> and password file <none>
I20260620 19:13:09.105664 28725 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:09.105715 28725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:09.106992 28725 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:09.107537 28741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:09.107686 28725 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:09.107770 28725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "14a18a72a17843da834e0042173e94a0"
format_stamp: "Formatted at 2026-06-20 19:13:08 on dist-test-slave-nq8r"
server_key: "d120cb97a52d9a47b295bd6fd9f22bf4"
server_key_iv: "8131d2b94a0467e07a24d05237417ecc"
server_key_version: "encryptionkey@0"
I20260620 19:13:09.108096 28725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:09.124976 28725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:09.126840 28725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:09.126953 28725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:09.127136 28725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:09.127403 28725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:09.127434 28725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:09.127454 28725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:09.127465 28725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:09.132717 28725 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:40449
I20260620 19:13:09.132788 28854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:40449 every 8 connection(s)
I20260620 19:13:09.133105 28725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.139644 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28725
W20260620 19:13:09.140069 28592 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.133764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:56983 (local address 127.19.66.254:33277)
0620 19:13:09.133825 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.133827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.134020 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:13:09.134514 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.134515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.134528 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.134593 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.135525 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.136015 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.137033 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.137223 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.137280 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.140008 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:56983: BlockingRecv error: recv got EOF from 127.19.66.195:56983 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:13:09.140125 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.133764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:56983)
0620 19:13:09.133904 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.133939 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.134416 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.134668 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.134671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.135233 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.135428 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.135432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.136102 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.136103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.136957 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.136960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.137291 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.137292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.137410 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.137855 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.137863 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.139914 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
W20260620 19:13:09.140450 28855 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:33277 (0 consecutive failures): Not authorized: Failed to ping master at 127.19.66.254:33277: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.145056 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.140667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:56205)
0620 19:13:09.140708 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.140716 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.140747 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.140942 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.140943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.141085 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.141139 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.141140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.141750 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.141751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.142261 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.142262 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.142672 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.142673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.142737 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.143161 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.143165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.144990 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 19:13:09.145390 28859 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.140709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:56205 (local address 127.19.66.254:33277)
0620 19:13:09.140813 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.140815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.140823 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:09.140861 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.140862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.140873 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.140938 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.141210 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.141679 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.142327 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.142610 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.142669 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.145059 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:56205: BlockingRecv error: recv got EOF from 127.19.66.195:56205 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":57,"threads_started":1}
I20260620 19:13:09.147398 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.140130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.19.66.254:33277)
0620 19:13:09.140198 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.140200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.140278 (+ 78us) server_negotiation.cc:407] Connection header received
0620 19:13:09.140346 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.140347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.140356 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.140393 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.141256 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.141735 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.142575 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.142798 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.142863 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.144143 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:09.144148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:09.144149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:09.144161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:09.146074 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:09.146431 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:09.146435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:09.146435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:09.146554 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:09.146724 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:09.146724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:09.146725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:09.147020 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:09.147160 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:09.147245 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:13:09.147327 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":51}
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.149396 28857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.145288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:45675)
0620 19:13:09.145319 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.145330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.145366 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.145560 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.145560 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.145681 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.145707 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.145708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.146359 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.146360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.147054 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.147057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.147556 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.147557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.147597 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.147940 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.147943 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.149345 (+ 1402us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 19:13:09.149485 28859 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.145365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:45675 (local address 127.19.66.254:33277)
0620 19:13:09.145467 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.145469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.145476 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:13:09.145494 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.145494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.145503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.145543 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.145766 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.146285 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.147125 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.147403 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.147467 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.149435 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:45675: BlockingRecv error: recv got EOF from 127.19.66.195:45675 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":97,"threads_started":1}
W20260620 19:13:09.149505 28855 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:09 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:09.904680 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:09.900326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:54667)
0620 19:13:09.900431 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:09.900442 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:09.900524 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:09.900648 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:09.900649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:09.900803 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:09.900890 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.900892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.901560 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.901561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.902104 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:09.902106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.902431 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.902432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:09.902491 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.902898 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:09.902903 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:09.904614 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
W20260620 19:13:09.904765 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:09.900409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:54667 (local address 127.19.66.254:33277)
0620 19:13:09.900507 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:09.900509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:09.900517 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:09.900572 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:09.900572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:09.900582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:09.900637 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:09.900982 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.901481 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.902179 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:09.902372 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:09.902425 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:09.904701 (+ 2276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:54667: BlockingRecv error: recv got EOF from 127.19.66.193:54667 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":45,"threads_started":1}
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:10.025315 28863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:10.020886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:43899)
0620 19:13:10.021035 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:10.021047 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:10.021128 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:10.021260 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:10.021261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:10.021435 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:10.021480 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.021482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.022117 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.022119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.022680 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.022682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.023000 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.023000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.023054 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.023474 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:10.023478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:10.025229 (+ 1751us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
W20260620 19:13:10.025389 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:10.020954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:43899 (local address 127.19.66.254:33277)
0620 19:13:10.021055 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:10.021057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:10.021119 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:13:10.021188 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:10.021189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:10.021201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:10.021248 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:10.021553 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.022048 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.022760 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.022943 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.022996 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.025339 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:43899: BlockingRecv error: recv got EOF from 127.19.66.194:43899 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:10.157974 28864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:10.149826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:35175)
0620 19:13:10.149939 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:10.149951 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:10.150014 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:10.150146 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:10.150147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:10.150331 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:10.150387 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.150390 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.151022 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.151023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.151577 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.151580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.151909 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.151909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.151961 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.156198 (+ 4237us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:10.156203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:10.157889 (+ 1686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
W20260620 19:13:10.158041 28862 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:10.149894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:35175 (local address 127.19.66.254:33277)
0620 19:13:10.149951 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:10.149953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:10.150003 (+ 50us) server_negotiation.cc:407] Connection header received
0620 19:13:10.150076 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:10.150076 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:10.150088 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:10.150130 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:10.150448 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.150953 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.151654 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.151851 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.151905 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.157985 (+ 6080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:35175: BlockingRecv error: recv got EOF from 127.19.66.195:35175 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:10 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:10.909444 28865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:10.905090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:51895)
0620 19:13:10.905215 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:10.905226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:10.905280 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:10.905394 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:10.905395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:10.905544 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:10.905595 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.905597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.906197 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.906198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.906742 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:10.906744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.907032 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.907032 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:10.907070 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.907524 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:10.907529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:10.909384 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20260620 19:13:10.909539 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:10.905177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:51895 (local address 127.19.66.254:33277)
0620 19:13:10.905269 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:10.905271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:10.905279 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:10.905324 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:10.905325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:10.905335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:10.905388 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:10.905645 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.906138 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.906800 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:10.906980 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:10.907024 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:10.909470 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:51895: BlockingRecv error: recv got EOF from 127.19.66.193:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":48,"threads_started":1}
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:11.029898 28867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:11.025739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:47775)
0620 19:13:11.025859 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:11.025869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:11.025924 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:11.026036 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:11.026036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:11.026186 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:11.026234 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.026236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.026839 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.026840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.027369 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.027370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.027650 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.027651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.027694 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.028110 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:11.028115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:11.029832 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
W20260620 19:13:11.030020 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:11.025806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:47775 (local address 127.19.66.254:33277)
0620 19:13:11.025855 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:11.025857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:11.025915 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:13:11.025967 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:11.025967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:11.025977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:11.026022 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:11.026283 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.026777 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.027422 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.027593 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.027633 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.029957 (+ 2324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:47775: BlockingRecv error: recv got EOF from 127.19.66.194:47775 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:11.162613 28868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:11.158388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:47021)
0620 19:13:11.158500 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:11.158511 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:11.158566 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:11.158692 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:11.158693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:11.158853 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:11.158898 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.158900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.159545 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.159546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.160077 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.160079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.160409 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.160410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.160465 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.160919 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:11.160924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:11.162555 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
W20260620 19:13:11.162699 28866 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:11.158501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:47021 (local address 127.19.66.254:33277)
0620 19:13:11.158561 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:11.158563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:11.158572 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:11.158613 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:11.158614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:11.158628 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:11.158670 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:11.158968 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.159472 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.160152 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.160350 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.160404 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.162636 (+ 2232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:47021: BlockingRecv error: recv got EOF from 127.19.66.195:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:11 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:11.914275 28869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:11.909900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:52505)
0620 19:13:11.910020 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:11.910029 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:11.910077 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:11.910237 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:11.910238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:11.910396 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:11.910442 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.910444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.911078 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.911079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.911631 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:11.911633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.911947 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.911948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:11.911985 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.912430 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:11.912435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:11.914197 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
W20260620 19:13:11.914348 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:11.909973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:52505 (local address 127.19.66.254:33277)
0620 19:13:11.910095 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:11.910097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:11.910107 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:11.910133 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:11.910133 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:11.910143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:11.910212 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:11.910507 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.911007 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.911693 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:11.911892 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:11.911944 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:11.914288 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:52505: BlockingRecv error: recv got EOF from 127.19.66.193:52505 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:12.034585 28871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:12.030371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:59285)
0620 19:13:12.030500 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:12.030510 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:12.030573 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:12.030717 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:12.030718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:12.030899 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:12.030945 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.030947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.031595 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.031596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.032117 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.032118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.032444 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.032445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.032481 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.032898 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:12.032904 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:12.034527 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260620 19:13:12.034664 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:12.030446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:59285 (local address 127.19.66.254:33277)
0620 19:13:12.030504 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:12.030506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:12.030583 (+ 77us) server_negotiation.cc:407] Connection header received
0620 19:13:12.030619 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:12.030619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:12.030634 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:12.030693 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:12.031010 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.031535 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.032185 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.032386 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.032444 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.034615 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:59285: BlockingRecv error: recv got EOF from 127.19.66.194:59285 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:12.167241 28872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:12.163054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:60937)
0620 19:13:12.163199 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:12.163210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:12.163269 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:12.163390 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:12.163391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:12.163541 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:12.163587 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.163589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.164203 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.164203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.164746 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.164747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.165101 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.165102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.165148 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.165542 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:12.165545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:12.167181 (+ 1636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
W20260620 19:13:12.167344 28870 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:12.163131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:60937 (local address 127.19.66.254:33277)
0620 19:13:12.163206 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:12.163208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:12.163257 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:13:12.163312 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:12.163313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:12.163324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:12.163379 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:12.163660 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.164143 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.164818 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.165039 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.165092 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.167272 (+ 2180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:60937: BlockingRecv error: recv got EOF from 127.19.66.195:60937 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:12 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:12.919209 28873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:12.914844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:41717)
0620 19:13:12.914967 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:12.914977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:12.915021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:12.915146 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:12.915147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:12.915301 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:12.915349 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.915351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.916001 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.916001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.916550 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:12.916552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.916911 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.916913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:12.916964 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.917399 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:12.917404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:12.919130 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
W20260620 19:13:12.919305 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:12.914898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:41717 (local address 127.19.66.254:33277)
0620 19:13:12.914998 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:12.914999 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:12.915022 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:13:12.915074 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:12.915074 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:12.915084 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:12.915131 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:12.915405 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.915941 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.916610 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:12.916809 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:12.916888 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:12.919245 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:41717: BlockingRecv error: recv got EOF from 127.19.66.193:41717 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":54,"threads_started":1}
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:13.039386 28875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:13.035072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:40337)
0620 19:13:13.035206 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:13.035218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:13.035276 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:13.035397 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:13.035397 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:13.035546 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:13.035594 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.035595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.036246 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.036248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.036878 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.036880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.037213 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.037214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.037262 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.037680 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:13.037685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:13.039325 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260620 19:13:13.039564 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:13.035146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:40337 (local address 127.19.66.254:33277)
0620 19:13:13.035225 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:13.035227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:13.035262 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:13:13.035327 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:13.035328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:13.035337 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:13.035386 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:13.035667 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.036162 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.036975 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.037152 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.037196 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.039454 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:40337: BlockingRecv error: recv got EOF from 127.19.66.194:40337 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:13.172075 28876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:13.167702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:37239)
0620 19:13:13.167845 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:13.167858 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:13.167933 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:13.168066 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:13.168067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:13.168245 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:13.168292 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.168293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.168992 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.168993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.169527 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.169530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.169855 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.169856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.169910 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.170337 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:13.170342 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:13.172019 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
W20260620 19:13:13.172181 28874 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:13.167796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:37239 (local address 127.19.66.254:33277)
0620 19:13:13.167861 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:13.167862 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:13.167922 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:13:13.167993 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:13.167994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:13.168007 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:13.168055 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:13.168361 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.168925 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.169604 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.169797 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.169851 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.172118 (+ 2267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:37239: BlockingRecv error: recv got EOF from 127.19.66.195:37239 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:13 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:13.924069 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:13.919748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:44699)
0620 19:13:13.919907 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:13.919918 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:13.919986 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:13.920118 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:13.920118 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:13.920317 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:13.920378 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.920380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.921066 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.921067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.921607 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:13.921609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.921938 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.921939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:13.921976 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.922437 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:13.922441 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:13.924012 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
W20260620 19:13:13.924183 28878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:13.919796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:44699 (local address 127.19.66.254:33277)
0620 19:13:13.919914 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:13.919916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:13.919978 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:13:13.920029 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:13.920030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:13.920040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:13.920108 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:13.920436 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.920994 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.921677 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:13.921885 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:13.921934 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:13.924103 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:44699: BlockingRecv error: recv got EOF from 127.19.66.193:44699 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":63,"threads_started":1}
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:14.044044 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:14.039837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:50523)
0620 19:13:14.039981 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:14.039993 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:14.040047 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:14.040165 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:14.040166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:14.040319 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:14.040366 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.040367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.040973 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.040973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.041503 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.041505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.041812 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.041813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.041849 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.042261 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:14.042266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:14.043984 (+ 1718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
W20260620 19:13:14.044147 28878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:14.039951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50523 (local address 127.19.66.254:33277)
0620 19:13:14.040023 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:14.040025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:14.040034 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:14.040092 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:14.040093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:14.040103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:14.040159 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:14.040420 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.040919 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.041568 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.041761 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.041809 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.044077 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:50523: BlockingRecv error: recv got EOF from 127.19.66.194:50523 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:14.176503 28880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:14.172509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:52079)
0620 19:13:14.172630 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:14.172640 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:14.172688 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:14.172804 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:14.172805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:14.172966 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:14.173013 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.173015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.173609 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.173609 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.174147 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.174148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.174425 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.174425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.174461 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.174858 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:14.174862 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:14.176445 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
W20260620 19:13:14.176595 28878 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:14.172590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:52079 (local address 127.19.66.254:33277)
0620 19:13:14.172674 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:14.172675 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:14.172685 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:14.172731 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:14.172731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:14.172743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:14.172793 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:14.173068 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.173553 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.174205 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.174377 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.174422 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.176526 (+ 2104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:52079: BlockingRecv error: recv got EOF from 127.19.66.195:52079 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:14 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:14.928685 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:14.924506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:47387)
0620 19:13:14.924640 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:14.924649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:14.924700 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:14.924863 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:14.924863 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:14.925033 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:14.925080 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.925082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.925691 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.925691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.926230 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:14.926231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.926515 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.926515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:14.926551 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.927002 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:14.927007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:14.928622 (+ 1615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
W20260620 19:13:14.928813 28882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:14.924596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:47387 (local address 127.19.66.254:33277)
0620 19:13:14.924717 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:14.924719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:14.924727 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:14.924763 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:14.924763 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:14.924773 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:14.924834 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:14.925138 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.925633 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.926285 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:14.926467 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:14.926511 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:14.928722 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:47387: BlockingRecv error: recv got EOF from 127.19.66.193:47387 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:15.048930 28883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:15.044508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:37407)
0620 19:13:15.044639 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:15.044648 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:15.044706 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:15.044836 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:15.044837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:15.045010 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:15.045060 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.045062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.045772 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.045773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.046332 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.046334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.046623 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.046623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.046660 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.047103 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:15.047110 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:15.048840 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
W20260620 19:13:15.049008 28882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:15.044539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37407 (local address 127.19.66.254:33277)
0620 19:13:15.044593 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:15.044595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:15.044701 (+ 106us) server_negotiation.cc:407] Connection header received
0620 19:13:15.044780 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:15.044780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:15.044791 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:15.044830 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:15.045113 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.045698 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.046392 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.046575 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.046619 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.048957 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:37407: BlockingRecv error: recv got EOF from 127.19.66.194:37407 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:15.185178 28884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:15.176955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:51477)
0620 19:13:15.177074 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:15.177087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:15.177149 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:15.177273 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:15.177273 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:15.177446 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:15.177487 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.177489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.178112 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.178113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.178625 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.178627 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.178940 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.178941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.178994 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.183386 (+ 4392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:15.183391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:15.185102 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260620 19:13:15.185276 28882 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:15.177014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:51477 (local address 127.19.66.254:33277)
0620 19:13:15.177079 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:15.177081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:15.177139 (+ 58us) server_negotiation.cc:407] Connection header received
0620 19:13:15.177205 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:15.177206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:15.177218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:15.177266 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:15.177551 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.178042 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.178697 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.178882 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.178935 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.185198 (+ 6263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:51477: BlockingRecv error: recv got EOF from 127.19.66.195:51477 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:15 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:15.933418 28885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:15.929113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:60987)
0620 19:13:15.929250 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:15.929260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:15.929313 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:15.929443 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:15.929444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:15.929598 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:15.929647 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.929649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.930276 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.930277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.930814 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:15.930815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.931090 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.931090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:15.931125 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.931560 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:15.931565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:15.933359 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20260620 19:13:15.933535 28886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:15.929199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:60987 (local address 127.19.66.254:33277)
0620 19:13:15.929330 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:15.929332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:15.929340 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:15.929362 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:15.929362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:15.929373 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:15.929432 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:15.929723 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.930221 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.930868 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:15.931045 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:15.931086 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:15.933454 (+ 2368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:60987: BlockingRecv error: recv got EOF from 127.19.66.193:60987 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:16.053866 28887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.049367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:44693)
0620 19:13:16.049487 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.049499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.049568 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.049698 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.049698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.049902 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.049947 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.049949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.050558 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.050559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.051144 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.051146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.051463 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.051464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.051515 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.051918 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.051922 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.053807 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
W20260620 19:13:16.053989 28886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:16.049432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:44693 (local address 127.19.66.254:33277)
0620 19:13:16.049490 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.049493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.049555 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:13:16.049635 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.049636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.049646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.049685 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.050010 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.050492 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.051218 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.051409 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.051460 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.053911 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:44693: BlockingRecv error: recv got EOF from 127.19.66.194:44693 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:16.189707 28888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.185611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:52673)
0620 19:13:16.185730 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.185741 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.185796 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.185910 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.185911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.186100 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.186150 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.186152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.186795 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.186795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.187322 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.187323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.187598 (+ 275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.187599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.187635 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.188037 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.188040 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.189648 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
W20260620 19:13:16.189828 28886 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:16.185680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:52673 (local address 127.19.66.254:33277)
0620 19:13:16.185749 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.185751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.185783 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:13:16.185835 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.185836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.185847 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.185900 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.186206 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.186733 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.187373 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.187553 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.187595 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.189758 (+ 2163us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:52673: BlockingRecv error: recv got EOF from 127.19.66.195:52673 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:16 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:16.938285 28889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:16.933872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:33417)
0620 19:13:16.934013 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:16.934027 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:16.934091 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:16.934312 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:16.934313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:16.934466 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:16.934516 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.934518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.935141 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.935142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.935651 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:16.935652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.936006 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.936007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:16.936061 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.936522 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:16.936526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:16.938226 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260620 19:13:16.938372 28890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:16.934007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:33417 (local address 127.19.66.254:33277)
0620 19:13:16.934187 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:16.934189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:16.934200 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:13:16.934224 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:16.934225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:16.934235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:16.934300 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:16.934573 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.935071 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.935710 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:16.935943 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:16.935992 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:16.938310 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:33417: BlockingRecv error: recv got EOF from 127.19.66.193:33417 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:17.058676 28891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:17.054297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:60509)
0620 19:13:17.054435 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:17.054446 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:17.054511 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:17.054647 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:17.054648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:17.054844 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:17.054891 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.054893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.055681 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.055683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.056203 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.056204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.056537 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.056538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.056607 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.057062 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:17.057066 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:17.058590 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260620 19:13:17.058740 28890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:17.054352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:60509 (local address 127.19.66.254:33277)
0620 19:13:17.054405 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:17.054407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:17.054507 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:13:17.054565 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:17.054566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:17.054577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:17.054626 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:17.054967 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.055612 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.056283 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.056479 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.056532 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.058690 (+ 2158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:60509: BlockingRecv error: recv got EOF from 127.19.66.194:60509 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:17.194597 28892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:17.190164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:52193)
0620 19:13:17.190321 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:17.190332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:17.190397 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:17.190527 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:17.190527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:17.190690 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:17.190734 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.190736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.191381 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.191382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.191901 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.191903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.192216 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.192217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.192269 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.192712 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:17.192716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:17.194523 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
W20260620 19:13:17.194736 28890 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:17.190278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:52193 (local address 127.19.66.254:33277)
0620 19:13:17.190340 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:17.190342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:17.190379 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:13:17.190453 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:17.190454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:17.190469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:17.190512 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:17.190824 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.191308 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.191980 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.192159 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.192212 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.194627 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:52193: BlockingRecv error: recv got EOF from 127.19.66.195:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:17 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:17.943015 28893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:17.938718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:40615)
0620 19:13:17.938834 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:17.938844 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:17.938895 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:17.938999 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:17.938999 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:17.939157 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:17.939220 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.939222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.939828 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.939829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.940379 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:17.940380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.940696 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.940697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:17.940740 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.941235 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:17.941239 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:17.942946 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
W20260620 19:13:17.943150 28894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:17.938793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:40615 (local address 127.19.66.254:33277)
0620 19:13:17.938892 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:17.938894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:17.938902 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:17.938938 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:17.938939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:17.938949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:17.938994 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:17.939273 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.939766 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.940446 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:17.940643 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:17.940692 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:17.943072 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:40615: BlockingRecv error: recv got EOF from 127.19.66.193:40615 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":50,"threads_started":1}
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:18.063357 28895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:18.059134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:38835)
0620 19:13:18.059257 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:18.059267 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:18.059319 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:18.059436 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:18.059437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:18.059590 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:18.059644 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.059645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.060239 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.060239 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.060773 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.060774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.061125 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.061126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.061179 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.061600 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:18.061604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:18.063297 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
W20260620 19:13:18.063454 28894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:18.059202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:38835 (local address 127.19.66.254:33277)
0620 19:13:18.059259 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:18.059260 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:18.059317 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:13:18.059364 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:18.059365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:18.059376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:18.059423 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:18.059697 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.060179 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.060829 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.061047 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.061096 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.063387 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:38835: BlockingRecv error: recv got EOF from 127.19.66.194:38835 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:18.199201 28896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:18.195004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:36541)
0620 19:13:18.195137 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:18.195150 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:18.195212 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:18.195327 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:18.195328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:18.195485 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:18.195530 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.195531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.196114 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.196115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.196691 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.196693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.197064 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.197065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.197127 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.197493 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:18.197497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:18.199134 (+ 1637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":53,"threads_started":1}
W20260620 19:13:18.199276 28894 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:18.195113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:36541 (local address 127.19.66.254:33277)
0620 19:13:18.195180 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:18.195182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:18.195195 (+ 13us) server_negotiation.cc:407] Connection header received
0620 19:13:18.195253 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:18.195254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:18.195265 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:18.195321 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:18.195577 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.196049 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.196748 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.196995 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.197051 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.199217 (+ 2166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:36541: BlockingRecv error: recv got EOF from 127.19.66.195:36541 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:18 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:18.947887 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:18.943422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:40349)
0620 19:13:18.943561 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:18.943574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:18.943626 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:18.943789 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:18.943790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:18.943958 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:18.944009 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.944011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.944739 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.944740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.945330 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:18.945333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.945674 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.945674 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:18.945711 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.946116 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:18.946120 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:18.947827 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260620 19:13:18.948001 28898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:18.943491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:40349 (local address 127.19.66.254:33277)
0620 19:13:18.943601 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:18.943603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:18.943613 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:18.943676 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:18.943677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:18.943687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:18.943776 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:18.944070 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.944656 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.945413 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:18.945610 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:18.945671 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:18.947922 (+ 2251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:40349: BlockingRecv error: recv got EOF from 127.19.66.193:40349 (error 108)
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:19.068281 28899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:19.063803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:56237)
0620 19:13:19.063929 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:19.063939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:19.063997 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:19.064132 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:19.064133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:19.064302 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:19.064349 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.064351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.065044 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.065045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.065571 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.065572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.065897 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.065898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.065947 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.066414 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:19.066419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:19.068193 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
W20260620 19:13:19.068368 28898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:19.063923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:56237 (local address 127.19.66.254:33277)
0620 19:13:19.063986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:19.063988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:19.063998 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:13:19.064061 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:19.064062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:19.064074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:19.064118 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:19.064419 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.064963 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.065648 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.065838 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.065910 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.068297 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:56237: BlockingRecv error: recv got EOF from 127.19.66.194:56237 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:19.204114 28900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:19.199706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:44059)
0620 19:13:19.199849 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:19.199928 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:19.199992 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:19.200127 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:19.200128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:19.200292 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:19.200357 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.200359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.200993 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.200993 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.201518 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.201519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.201875 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.201876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.201911 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.202333 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:19.202337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:19.204039 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":53,"threads_started":1}
W20260620 19:13:19.204183 28898 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:19.199832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:44059 (local address 127.19.66.254:33277)
0620 19:13:19.199885 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:19.199887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:19.199992 (+ 105us) server_negotiation.cc:407] Connection header received
0620 19:13:19.200049 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:19.200050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:19.200063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:19.200112 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:19.200414 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.200943 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.201593 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.201820 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.201873 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.204131 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:44059: BlockingRecv error: recv got EOF from 127.19.66.195:44059 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:19 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:19.952896 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:19.948304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:58361)
0620 19:13:19.948432 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:19.948442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:19.948498 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:19.948617 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:19.948618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:19.948817 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:19.948907 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.948910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.949586 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.949587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.950172 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:19.950175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.950525 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.950527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:19.950595 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.951003 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:19.951007 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:19.952777 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
W20260620 19:13:19.952961 28902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:19.948379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:58361 (local address 127.19.66.254:33277)
0620 19:13:19.948481 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:19.948483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:19.948491 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:19.948542 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:19.948542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:19.948552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:19.948607 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:19.948978 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.949525 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.950230 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:19.950444 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:19.950494 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:19.952901 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:58361: BlockingRecv error: recv got EOF from 127.19.66.193:58361 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":45,"threads_started":1}
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:20.073531 28903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:20.068911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:46683)
0620 19:13:20.069054 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:20.069065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:20.069135 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:20.069257 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:20.069258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:20.069418 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:20.069462 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.069464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.070144 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.070145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.070824 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.070826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.071169 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.071169 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.071211 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.071699 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:20.071704 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:20.073457 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
W20260620 19:13:20.073616 28902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:20.068980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:46683 (local address 127.19.66.254:33277)
0620 19:13:20.069084 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:20.069087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:20.069152 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:13:20.069179 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:20.069180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:20.069190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:20.069244 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:20.069530 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.070026 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.070889 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.071090 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.071137 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.073555 (+ 2418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:46683: BlockingRecv error: recv got EOF from 127.19.66.194:46683 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:20.212272 28904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:20.204655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:48153)
0620 19:13:20.204784 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:20.204797 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:20.204875 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:20.205002 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:20.205003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:20.205175 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:20.205221 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.205222 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.205842 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.205843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.206381 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.206383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.206732 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.206733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.206776 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.210339 (+ 3563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:20.210344 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:20.212172 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
W20260620 19:13:20.212339 28902 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:20.204754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:48153 (local address 127.19.66.254:33277)
0620 19:13:20.204829 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:20.204831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:20.204842 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:13:20.204934 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:20.204935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:20.204949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:20.204992 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:20.205277 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.205772 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.206473 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.206665 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.206722 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.212267 (+ 5545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:48153: BlockingRecv error: recv got EOF from 127.19.66.195:48153 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:20 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:20.957716 28905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:20.953297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:45475)
0620 19:13:20.953434 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:20.953445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:20.953498 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:20.953658 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:20.953659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:20.953859 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:20.953906 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.953908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.954556 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.954557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.955140 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:20.955142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.955479 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.955480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:20.955518 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.955940 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:20.955945 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:20.957657 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
W20260620 19:13:20.957834 28906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:20.953395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:45475 (local address 127.19.66.254:33277)
0620 19:13:20.953503 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:20.953504 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:20.953531 (+ 27us) server_negotiation.cc:407] Connection header received
0620 19:13:20.953552 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:20.953552 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:20.953562 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:20.953610 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:20.953987 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.954485 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.955215 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:20.955419 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:20.955469 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:20.957751 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:45475: BlockingRecv error: recv got EOF from 127.19.66.193:45475 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:21.078409 28907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:21.074041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:40231)
0620 19:13:21.074174 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:21.074184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:21.074241 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:21.074379 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:21.074380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:21.074543 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:21.074585 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.074587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.075190 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.075191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.075719 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.075720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.076008 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.076009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.076047 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.076551 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:21.076556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:21.078332 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
W20260620 19:13:21.078495 28906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:21.074127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:40231 (local address 127.19.66.254:33277)
0620 19:13:21.074178 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:21.074179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:21.074238 (+ 59us) server_negotiation.cc:407] Connection header received
0620 19:13:21.074291 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:21.074292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:21.074302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:21.074363 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:21.074638 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.075124 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.075777 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.075955 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.076000 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.078433 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:40231: BlockingRecv error: recv got EOF from 127.19.66.194:40231 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:21.217315 28908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:21.212817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:34985)
0620 19:13:21.212969 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:21.212979 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:21.213041 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:21.213168 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:21.213168 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:21.213331 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:21.213378 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.213380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.213996 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.213997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.214516 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.214518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.214821 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.214822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.214883 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.215375 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:21.215381 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:21.217243 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
W20260620 19:13:21.217397 28906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:21.212889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:34985 (local address 127.19.66.254:33277)
0620 19:13:21.212941 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:21.212943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:21.213031 (+ 88us) server_negotiation.cc:407] Connection header received
0620 19:13:21.213099 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:21.213100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:21.213114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:21.213157 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:21.213443 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.213927 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.214584 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.214764 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.214817 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.217333 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:34985: BlockingRecv error: recv got EOF from 127.19.66.195:34985 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:21 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:21.962246 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:21.958129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:36327)
0620 19:13:21.958238 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:21.958249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:21.958297 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:21.958421 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:21.958422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:21.958575 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:21.958622 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.958624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.959240 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.959241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.959787 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:21.959789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.960074 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.960075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:21.960109 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.960509 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:21.960514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:21.962156 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
W20260620 19:13:21.962318 28910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:21.958185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:36327 (local address 127.19.66.254:33277)
0620 19:13:21.958312 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:21.958314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:21.958322 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:21.958342 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:21.958343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:21.958352 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:21.958411 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:21.958675 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.959189 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.959843 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:21.960028 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:21.960070 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:21.962253 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:36327: BlockingRecv error: recv got EOF from 127.19.66.193:36327 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:22.083546 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:22.078957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:41603)
0620 19:13:22.079094 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:22.079103 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:22.079153 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:22.079294 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:22.079295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:22.079449 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:22.079492 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.079494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.080317 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.080319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.080890 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.080893 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.081230 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.081231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.081285 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.081723 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:22.081728 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:22.083455 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
W20260620 19:13:22.083608 28910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:22.079108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:41603 (local address 127.19.66.254:33277)
0620 19:13:22.079177 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:22.079179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:22.079187 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:22.079211 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:22.079213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:22.079222 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:22.079277 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:22.079577 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.080250 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.080957 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.081173 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.081228 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.083555 (+ 2327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:41603: BlockingRecv error: recv got EOF from 127.19.66.194:41603 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:22.222448 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:22.217905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:49491)
0620 19:13:22.218029 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:22.218039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:22.218098 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:22.218241 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:22.218242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:22.218450 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:22.218505 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.218508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.219188 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.219189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.219781 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.219783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.220103 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.220104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.220155 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.220615 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:22.220619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:22.222379 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260620 19:13:22.222518 28910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:22.217963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:49491 (local address 127.19.66.254:33277)
0620 19:13:22.218032 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:22.218033 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:22.218098 (+ 65us) server_negotiation.cc:407] Connection header received
0620 19:13:22.218142 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:22.218143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:22.218157 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:22.218222 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:22.218582 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.219118 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.219844 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.220023 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.220068 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.222461 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:49491: BlockingRecv error: recv got EOF from 127.19.66.195:49491 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:22 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:22.967007 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:22.962723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:49289)
0620 19:13:22.962845 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:22.962856 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:22.962908 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:22.963024 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:22.963025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:22.963196 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:22.963245 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.963247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.963843 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.963843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.964380 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:22.964382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.964666 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.964667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:22.964702 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.965200 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:22.965205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:22.966949 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
W20260620 19:13:22.967106 28914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:22.962794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:49289 (local address 127.19.66.254:33277)
0620 19:13:22.962902 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:22.962904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:22.962913 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:22.962956 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:22.962956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:22.962966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:22.963011 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:22.963299 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.963791 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.964436 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:22.964618 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:22.964662 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:22.967035 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:49289: BlockingRecv error: recv got EOF from 127.19.66.193:49289 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":47,"threads_started":1}
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:23.088289 28915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:23.084052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:59805)
0620 19:13:23.084204 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:23.084216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:23.084285 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:23.084432 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:23.084433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:23.084593 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:23.084639 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.084640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.085266 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.085266 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.085791 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.085793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.086118 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.086119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.086166 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.086575 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:23.086579 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:23.088211 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":70,"threads_started":1}
W20260620 19:13:23.088372 28914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:23.084142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:59805 (local address 127.19.66.254:33277)
0620 19:13:23.084203 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:23.084205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:23.084282 (+ 77us) server_negotiation.cc:407] Connection header received
0620 19:13:23.084365 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:23.084366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:23.084377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:23.084416 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:23.084690 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.085204 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.085852 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.086063 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.086110 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.088312 (+ 2202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:59805: BlockingRecv error: recv got EOF from 127.19.66.194:59805 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.195@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:23.227032 28916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:23.222865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.195:38281)
0620 19:13:23.222998 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:23.223007 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:23.223064 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:23.223185 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:23.223186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:23.223346 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:23.223394 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.223396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.223983 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.223984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.224526 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.224527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.224795 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.224796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.224834 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.225304 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:23.225308 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:23.226974 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
W20260620 19:13:23.227120 28914 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:23.222915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:38281 (local address 127.19.66.254:33277)
0620 19:13:23.222959 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:23.222961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:23.223060 (+ 99us) server_negotiation.cc:407] Connection header received
0620 19:13:23.223116 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:23.223116 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:23.223128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:23.223166 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:23.223443 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.223923 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.224582 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.224749 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.224788 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.227060 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.195:38281: BlockingRecv error: recv got EOF from 127.19.66.195:38281 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:23 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.193@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:23.971709 28917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:23.967506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.193:59797)
0620 19:13:23.967612 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:23.967621 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:23.967712 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:23.967833 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:23.967833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:23.967985 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:23.968034 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.968036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.968638 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.968639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.969219 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:23.969220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.969515 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.969515 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:23.969550 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.969990 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:23.969994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:23.971634 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
W20260620 19:13:23.971789 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:23.967567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:59797 (local address 127.19.66.254:33277)
0620 19:13:23.967659 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:23.967661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:23.967669 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:23.967762 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:23.967763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:23.967774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:23.967826 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:23.968080 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.968571 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.969276 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:23.969465 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:23.969512 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:23.971728 (+ 2216us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.193:59797: BlockingRecv error: recv got EOF from 127.19.66.193:59797 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":45,"threads_started":1}
Jun 20 19:13:24 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 19:13:24 dist-test-slave-nq8r krb5kdc[27858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.19.66.194@KRBTEST.COM for oryx/127.19.66.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 19:13:24.093079 28919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.088814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:33277 (local address 127.19.66.194:37585)
0620 19:13:24.088963 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:24.088973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:24.089030 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:24.089167 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:24.089167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:24.089331 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:24.089376 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.089377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.090048 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.090049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.090610 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.090612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.090929 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.090930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.090979 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.091407 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:24.091412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:24.093011 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.19.66.254:33277: Server oryx/127.19.66.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260620 19:13:24.093166 28918 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:24.088934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37585 (local address 127.19.66.254:33277)
0620 19:13:24.088985 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.088986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.089017 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:13:24.089082 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.089083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.089093 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.089149 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.089458 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.089956 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.090675 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.090854 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.090903 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.093100 (+ 2197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.19.66.194:37585: BlockingRecv error: recv got EOF from 127.19.66.194:37585 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:13:24.147204 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28441
I20260620 19:13:24.151203 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28593
I20260620 19:13:24.154953 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28725
I20260620 19:13:24.158535 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28371
2026-06-20T19:13:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17234 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T19:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:24Z Disabled control of system clock
I20260620 19:13:24.184134 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35331
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 19:13:24.261092 28923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.261286 28923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.261304 28923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.262466 28923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:24.262507 28923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.262518 28923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.262529 28923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:24.262548 28923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:24.263875 28923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.264093 28923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.264366 28923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.268725 28931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.268795 28929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.268916 28928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.268761 28923 server_base.cc:1034] running on GCE node
I20260620 19:13:24.269187 28923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.269447 28923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.270587 28923 hybrid_clock.cc:648] HybridClock initialized: now 1781982804270567 us; error 34 us; skew 500 ppm
I20260620 19:13:24.271678 28923 webserver.cc:460] Webserver started at http://127.19.66.254:45403/ using document root <none> and password file <none>
I20260620 19:13:24.271911 28923 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.271958 28923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.272080 28923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.273103 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "b299f88878a5422fa7004299e84d194f"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "746cac7818785e58888bfc2e11d94ec4"
server_key_iv: "38c9d8c46c3f6114addc8334b28ecca0"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.273413 28923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "b299f88878a5422fa7004299e84d194f"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "746cac7818785e58888bfc2e11d94ec4"
server_key_iv: "38c9d8c46c3f6114addc8334b28ecca0"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.274513 28923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:24.275301 28937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.275475 28923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:24.275559 28923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "b299f88878a5422fa7004299e84d194f"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "746cac7818785e58888bfc2e11d94ec4"
server_key_iv: "38c9d8c46c3f6114addc8334b28ecca0"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.275624 28923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.288071 28923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.288334 28923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.288451 28923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.292349 28923 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:35331
I20260620 19:13:24.292375 28989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:35331 every 8 connection(s)
I20260620 19:13:24.292677 28923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:24.293313 28990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:24.295729 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f: Bootstrap starting.
I20260620 19:13:24.296259 28990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:24.296517 28990 log.cc:824] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:24.297106 28990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f: No bootstrap required, opened a new log
I20260620 19:13:24.297804 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 28923
I20260620 19:13:24.297894 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:13:24.298074 19723 external_mini_cluster.cc:1428] Setting key 5e46865232527472a2a1d6043bf364ee
I20260620 19:13:24.299000 28990 raft_consensus.cc:348] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } }
I20260620 19:13:24.299139 28990 raft_consensus.cc:374] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:24.299171 28990 raft_consensus.cc:729] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b299f88878a5422fa7004299e84d194f, State: Initialized, Role: FOLLOWER
I20260620 19:13:24.299328 28990 consensus_queue.cc:260] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } }
I20260620 19:13:24.299412 28990 raft_consensus.cc:388] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:24.299433 28990 raft_consensus.cc:482] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:24.299467 28990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:24.299925 28990 raft_consensus.cc:504] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } }
I20260620 19:13:24.300001 28990 leader_election.cc:302] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b299f88878a5422fa7004299e84d194f; no voters:
I20260620 19:13:24.300133 28990 leader_election.cc:288] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:24.300184 28995 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:24.300349 28990 sys_catalog.cc:564] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:24.300383 28995 raft_consensus.cc:686] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [term 1 LEADER]: Becoming Leader. State: Replica: b299f88878a5422fa7004299e84d194f, State: Running, Role: LEADER
I20260620 19:13:24.300593 28995 consensus_queue.cc:237] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } }
I20260620 19:13:24.300976 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b299f88878a5422fa7004299e84d194f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } } }
I20260620 19:13:24.301052 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:24.301175 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b299f88878a5422fa7004299e84d194f. Latest consensus state: current_term: 1 leader_uuid: "b299f88878a5422fa7004299e84d194f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b299f88878a5422fa7004299e84d194f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35331 } } }
I20260620 19:13:24.301214 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:24.301357 29004 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:24.301848 29004 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:24.303579 29004 catalog_manager.cc:1347] Generated new cluster ID: 46a7b86b008b409eb5c54b09c75de84b
I20260620 19:13:24.303622 29004 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:24.304092 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.298537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.19.66.254:35331)
0620 19:13:24.298712 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.298719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.298742 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:13:24.298961 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.298966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.299023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.299126 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.299361 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.300835 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.301971 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.302562 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.302674 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.303437 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.303442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.303442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.303629 (+ 187us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.303668 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.303684 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.303751 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.303823 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.303897 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260620 19:13:24.313457 29004 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:24.313906 29004 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:24.327791 29004 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b299f88878a5422fa7004299e84d194f: Generated new TSK 0
I20260620 19:13:24.327960 29004 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:13:24.332818 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--builtin_ntp_servers=127.19.66.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:13:24.411414 29014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.411626 29014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.411653 29014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.412890 29014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.412930 29014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.412961 29014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:24.414297 29014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.414538 29014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.414810 29014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.419440 29019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.419553 29022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.419575 29020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.419920 29014 server_base.cc:1034] running on GCE node
I20260620 19:13:24.420130 29014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.420377 29014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.421552 29014 hybrid_clock.cc:648] HybridClock initialized: now 1781982804421539 us; error 29 us; skew 500 ppm
I20260620 19:13:24.422662 29014 webserver.cc:460] Webserver started at http://127.19.66.193:36669/ using document root <none> and password file <none>
I20260620 19:13:24.422859 29014 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.422899 29014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.422991 29014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.423983 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "40feaa0733de44f683c6b0fab1e1a824"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "6997739edc0035fe6ac0b243c8cf604e"
server_key_iv: "45901ad5b20ad8b2e70f9085bcc2bfc9"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.424259 29014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "40feaa0733de44f683c6b0fab1e1a824"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "6997739edc0035fe6ac0b243c8cf604e"
server_key_iv: "45901ad5b20ad8b2e70f9085bcc2bfc9"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.425457 29014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:24.426090 29028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.426260 29014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:24.426337 29014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "40feaa0733de44f683c6b0fab1e1a824"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "6997739edc0035fe6ac0b243c8cf604e"
server_key_iv: "45901ad5b20ad8b2e70f9085bcc2bfc9"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.426403 29014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.443835 29014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.444192 29014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.444337 29014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.444608 29014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:24.445036 29014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:24.445073 29014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.445093 29014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:24.445106 29014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.450289 29014 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:45447
I20260620 19:13:24.450357 29141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:45447 every 8 connection(s)
I20260620 19:13:24.450654 29014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:24.455307 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.451312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:47055 (local address 127.19.66.254:35331)
0620 19:13:24.451389 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.451392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.451588 (+ 196us) server_negotiation.cc:407] Connection header received
0620 19:13:24.452055 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.452056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.452067 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.452114 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.452711 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.453210 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.454139 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.454327 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.454379 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.455070 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.455072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.455072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.455076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.455090 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.455094 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.455145 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.455208 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.455236 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 19:13:24.455497 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.451288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35331 (local address 127.19.66.193:47055)
0620 19:13:24.451473 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:24.451507 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:24.451976 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:24.452174 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:24.452177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:24.452185 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:24.452636 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.452639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.453270 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.453271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.454060 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.454062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.454386 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.454387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.454486 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.454945 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:24.454945 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:24.454972 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:24.454993 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:24.455144 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:24.455147 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:24.455149 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:13:24.455202 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 19:13:24.455299 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260620 19:13:24.455958 29142 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35331
I20260620 19:13:24.456045 29142 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:24.456216 29142 heartbeater.cc:511] Master 127.19.66.254:35331 requested a full tablet report, sending...
I20260620 19:13:24.456681 28954 ts_manager.cc:194] Registered new tserver with Master: 40feaa0733de44f683c6b0fab1e1a824 (127.19.66.193:45447)
I20260620 19:13:24.457317 28954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.193:47055
I20260620 19:13:24.457522 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29014
I20260620 19:13:24.457605 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:24.457780 19723 external_mini_cluster.cc:1428] Setting key 43bd59b4f62a1fd440ea9869e2e54a64
I20260620 19:13:24.458539 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--builtin_ntp_servers=127.19.66.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:13:24.536136 29145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.536329 29145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.536348 29145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.537520 29145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.537559 29145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.537582 29145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:24.538863 29145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.539063 29145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.539379 29145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.543887 29151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.543951 29153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.543972 29150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.544239 29145 server_base.cc:1034] running on GCE node
I20260620 19:13:24.544436 29145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.544631 29145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.545763 29145 hybrid_clock.cc:648] HybridClock initialized: now 1781982804545737 us; error 40 us; skew 500 ppm
I20260620 19:13:24.546967 29145 webserver.cc:460] Webserver started at http://127.19.66.194:37921/ using document root <none> and password file <none>
I20260620 19:13:24.547194 29145 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.547240 29145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.547375 29145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.548418 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "aa3d8ee35cbe4017abaf670895a2465a"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "3ca13769036e510e89f8ce123223d3cf"
server_key_iv: "0e6d3e502282b84de4ef55b4aa242bbc"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.548727 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "aa3d8ee35cbe4017abaf670895a2465a"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "3ca13769036e510e89f8ce123223d3cf"
server_key_iv: "0e6d3e502282b84de4ef55b4aa242bbc"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.549893 29145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:13:24.550522 29159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.550650 29145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:13:24.550720 29145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "aa3d8ee35cbe4017abaf670895a2465a"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "3ca13769036e510e89f8ce123223d3cf"
server_key_iv: "0e6d3e502282b84de4ef55b4aa242bbc"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.550781 29145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.560148 29145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.560448 29145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.560567 29145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.560760 29145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:24.561086 29145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:24.561120 29145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.561147 29145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:24.561173 29145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.566406 29145 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:44533
I20260620 19:13:24.566457 29272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:44533 every 8 connection(s)
I20260620 19:13:24.566749 29145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:24.571570 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.567422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:58861 (local address 127.19.66.254:35331)
0620 19:13:24.567474 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.567476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.567687 (+ 211us) server_negotiation.cc:407] Connection header received
0620 19:13:24.568171 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.568173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.568184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.568241 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.568906 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.569401 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.570317 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.570511 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.570565 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.571321 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.571322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.571323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.571328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.571343 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.571347 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.571392 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.571465 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.571494 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:13:24.571772 29275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.567419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35331 (local address 127.19.66.194:58861)
0620 19:13:24.567577 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:24.567610 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:24.568079 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:24.568326 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:24.568330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:24.568338 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:24.568803 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.568808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.569478 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.569479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.570232 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.570234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.570577 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.570578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.570696 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.571168 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:24.571169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:24.571195 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:24.571217 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:24.571397 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:24.571402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:24.571405 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:24.571465 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:13:24.571565 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260620 19:13:24.572183 29273 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35331
I20260620 19:13:24.572269 29273 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:24.572441 29273 heartbeater.cc:511] Master 127.19.66.254:35331 requested a full tablet report, sending...
I20260620 19:13:24.572480 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29145
I20260620 19:13:24.572552 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:24.572737 19723 external_mini_cluster.cc:1428] Setting key 168b1d4329447b24a3d2e4381809f9e5
I20260620 19:13:24.572793 28954 ts_manager.cc:194] Registered new tserver with Master: aa3d8ee35cbe4017abaf670895a2465a (127.19.66.194:44533)
I20260620 19:13:24.573285 28954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.194:58861
I20260620 19:13:24.573733 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--builtin_ntp_servers=127.19.66.212:41915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 19:13:24.650525 29276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.650722 29276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.650739 29276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.651839 29276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.651890 29276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.651914 29276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:24.653270 29276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:41915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.653489 29276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.653764 29276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.658536 29284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.658564 29282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.658638 29276 server_base.cc:1034] running on GCE node
W20260620 19:13:24.658588 29281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.658938 29276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.659171 29276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.660311 29276 hybrid_clock.cc:648] HybridClock initialized: now 1781982804660290 us; error 36 us; skew 500 ppm
I20260620 19:13:24.661520 29276 webserver.cc:460] Webserver started at http://127.19.66.195:37681/ using document root <none> and password file <none>
I20260620 19:13:24.661721 29276 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.661768 29276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.661926 29276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.662963 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "db7271b6107a4804a02ba81d1bbcc8ed"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "ca7640ba8bdee9760283c01d15675c8b"
server_key_iv: "2d3db04af680dec25a37305872536e33"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.663280 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "db7271b6107a4804a02ba81d1bbcc8ed"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "ca7640ba8bdee9760283c01d15675c8b"
server_key_iv: "2d3db04af680dec25a37305872536e33"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.664425 29276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:24.665108 29290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.665247 29276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:13:24.665323 29276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "db7271b6107a4804a02ba81d1bbcc8ed"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "ca7640ba8bdee9760283c01d15675c8b"
server_key_iv: "2d3db04af680dec25a37305872536e33"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.665385 29276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.676702 29276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.676997 29276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.677124 29276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.677314 29276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:24.677637 29276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:24.677680 29276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.677716 29276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:24.677765 29276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.683084 29276 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:38853
I20260620 19:13:24.683133 29403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:38853 every 8 connection(s)
I20260620 19:13:24.683441 29276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:24.687888 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29276
I20260620 19:13:24.687983 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:24.688053 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.684039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:41169 (local address 127.19.66.254:35331)
0620 19:13:24.684083 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.684085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.684306 (+ 221us) server_negotiation.cc:407] Connection header received
0620 19:13:24.684760 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.684761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.684772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.684834 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.685496 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.685983 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.686879 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.687053 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.687095 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.687779 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.687780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.687781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.687787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.687827 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.687832 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.687893 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.687946 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.687980 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:24.688156 19723 external_mini_cluster.cc:1428] Setting key e05c6a90a1f4c35c28a9ea373f4d76a1
I20260620 19:13:24.688262 29406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.684033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35331 (local address 127.19.66.195:41169)
0620 19:13:24.684190 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:24.684222 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:24.684681 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:24.684926 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:24.684930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:24.684941 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:24.685431 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.685435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.686046 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.686046 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.686822 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:24.686824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:24.687104 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:24.687104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:24.687204 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:24.687649 (+ 445us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:24.687650 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:24.687676 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:24.687696 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:24.687895 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:24.687898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:24.687900 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:13:24.687944 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:13:24.688046 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260620 19:13:24.688699 29404 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35331
I20260620 19:13:24.688781 29404 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:24.688992 29404 heartbeater.cc:511] Master 127.19.66.254:35331 requested a full tablet report, sending...
I20260620 19:13:24.689304 28954 ts_manager.cc:194] Registered new tserver with Master: db7271b6107a4804a02ba81d1bbcc8ed (127.19.66.195:38853)
I20260620 19:13:24.689657 28954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.195:41169
I20260620 19:13:24.699966 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:13:24.701644 28994 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:24.701199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.19.66.254:35331)
0620 19:13:24.701253 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.701255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.701365 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:13:24.701425 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.701426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.701437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.701472 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.701575 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:13:24.701709 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29014
I20260620 19:13:24.705674 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29145
I20260620 19:13:24.709676 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29276
I20260620 19:13:24.713088 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 28923
2026-06-20T19:13:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T19:13:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:24Z Disabled control of system clock
I20260620 19:13:24.737584 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37399
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:37399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:13:24.814771 29415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.814961 29415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.814980 29415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.816090 29415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:24.816131 29415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.816143 29415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.816154 29415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:24.816164 29415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:24.817512 29415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:37399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.817740 29415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.817997 29415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.822414 29420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.822436 29423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.822508 29415 server_base.cc:1034] running on GCE node
W20260620 19:13:24.822618 29421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.822849 29415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.823087 29415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.824229 29415 hybrid_clock.cc:648] HybridClock initialized: now 1781982804824206 us; error 36 us; skew 500 ppm
I20260620 19:13:24.825470 29415 webserver.cc:460] Webserver started at http://127.19.66.254:33599/ using document root <none> and password file <none>
I20260620 19:13:24.825683 29415 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.825732 29415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.825858 29415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.826838 29415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "ad44ab5942cd4b01a1ba35d96cf854db"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "84a5b15933a0f3c43eff8cb85d9e8fb9"
server_key_iv: "9cb533ee120abe5d75fac3c207a8e316"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.827144 29415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "ad44ab5942cd4b01a1ba35d96cf854db"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "84a5b15933a0f3c43eff8cb85d9e8fb9"
server_key_iv: "9cb533ee120abe5d75fac3c207a8e316"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.828284 29415 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:13:24.828917 29429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.829046 29415 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.829123 29415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "ad44ab5942cd4b01a1ba35d96cf854db"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "84a5b15933a0f3c43eff8cb85d9e8fb9"
server_key_iv: "9cb533ee120abe5d75fac3c207a8e316"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.829188 29415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.832342 29415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.832561 29415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.832671 29415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.836084 29415 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:37399
I20260620 19:13:24.836117 29481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:37399 every 8 connection(s)
I20260620 19:13:24.836413 29415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:24.837066 29482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:24.839637 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db: Bootstrap starting.
I20260620 19:13:24.840183 29482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:24.840412 29482 log.cc:824] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:24.841089 29482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db: No bootstrap required, opened a new log
I20260620 19:13:24.841167 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29415
I20260620 19:13:24.841254 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:13:24.841415 19723 external_mini_cluster.cc:1428] Setting key ae8f9b73198ad9ee14d5a69277b4a593
I20260620 19:13:24.842968 29482 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } }
I20260620 19:13:24.843106 29482 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:24.843137 29482 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad44ab5942cd4b01a1ba35d96cf854db, State: Initialized, Role: FOLLOWER
I20260620 19:13:24.843288 29482 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } }
I20260620 19:13:24.843371 29482 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:24.843403 29482 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:24.843464 29482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:24.843971 29482 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } }
I20260620 19:13:24.844101 29482 leader_election.cc:302] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad44ab5942cd4b01a1ba35d96cf854db; no voters:
I20260620 19:13:24.844249 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.841916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58416 (local address 127.19.66.254:37399)
0620 19:13:24.842101 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.842108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.842132 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:13:24.842941 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.842946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.842990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.843108 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.843789 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.843793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.843793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.843797 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.843841 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.843852 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.843908 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.843972 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.844027 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260620 19:13:24.844280 29482 leader_election.cc:288] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:24.844515 29482 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:24.844548 29487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:24.844652 29487 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [term 1 LEADER]: Becoming Leader. State: Replica: ad44ab5942cd4b01a1ba35d96cf854db, State: Running, Role: LEADER
I20260620 19:13:24.844799 29487 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } }
I20260620 19:13:24.845203 29487 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad44ab5942cd4b01a1ba35d96cf854db. Latest consensus state: current_term: 1 leader_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } } }
I20260620 19:13:24.845278 29487 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:24.845363 29489 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44ab5942cd4b01a1ba35d96cf854db" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37399 } } }
I20260620 19:13:24.845513 29489 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:24.845721 29496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:24.846185 29496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:24.847530 29496 catalog_manager.cc:1347] Generated new cluster ID: 41e63fd0f9654845b91f1bea45db9ba7
I20260620 19:13:24.847579 29496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:24.866128 29496 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:24.866321 29496 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:24.870476 29496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad44ab5942cd4b01a1ba35d96cf854db: Generated new TSK 0
I20260620 19:13:24.870658 29496 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:13:24.872046 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--builtin_ntp_servers=127.19.66.212:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:13:24.952870 29506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:24.953097 29506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:24.953130 29506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:24.954334 29506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:24.954376 29506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:24.954412 29506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:24.955766 29506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:24.956014 29506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:24.956333 29506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:24.962142 29514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.962149 29512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:24.962121 29511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:24.962379 29506 server_base.cc:1034] running on GCE node
I20260620 19:13:24.962595 29506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:24.962821 29506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:24.963970 29506 hybrid_clock.cc:648] HybridClock initialized: now 1781982804963958 us; error 26 us; skew 500 ppm
I20260620 19:13:24.965242 29506 webserver.cc:460] Webserver started at http://127.19.66.193:36009/ using document root <none> and password file <none>
I20260620 19:13:24.965471 29506 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:24.965518 29506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:24.965638 29506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:24.966686 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "1801cdd35139484b807bf0cae1d0d7ac"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "c4f19262b2560b8e55d82ae6f10f1860"
server_key_iv: "81757335cc99d61f8655aefd2537159d"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.967015 29506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "1801cdd35139484b807bf0cae1d0d7ac"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "c4f19262b2560b8e55d82ae6f10f1860"
server_key_iv: "81757335cc99d61f8655aefd2537159d"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.968212 29506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:13:24.968926 29520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.969084 29506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.969163 29506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "1801cdd35139484b807bf0cae1d0d7ac"
format_stamp: "Formatted at 2026-06-20 19:13:24 on dist-test-slave-nq8r"
server_key: "c4f19262b2560b8e55d82ae6f10f1860"
server_key_iv: "81757335cc99d61f8655aefd2537159d"
server_key_version: "encryptionkey@0"
I20260620 19:13:24.969228 29506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:24.972433 29506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:24.972681 29506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:24.972797 29506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:24.972999 29506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:24.973335 29506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:24.973376 29506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.973400 29506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:24.973416 29506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:24.979848 29506 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:44973
I20260620 19:13:24.979898 29633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:44973 every 8 connection(s)
I20260620 19:13:24.980257 29506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:13:24.983631 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.981197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:50693 (local address 127.19.66.254:37399)
0620 19:13:24.981245 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:24.981247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:24.981599 (+ 352us) server_negotiation.cc:407] Connection header received
0620 19:13:24.982240 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:24.982242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:24.982277 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:24.982362 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:24.983452 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:24.983453 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:24.983453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:24.983457 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:24.983471 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:24.983475 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:24.983519 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:24.983566 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:13:24.983578 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:13:24.983855 29636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:24.981256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37399 (local address 127.19.66.193:50693)
0620 19:13:24.981465 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:24.981509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:24.982136 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:24.982444 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:24.982448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:24.982456 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:24.983313 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:24.983313 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:24.983342 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:24.983369 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:24.983516 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:24.983518 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:24.983521 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:24.983565 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:13:24.983611 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260620 19:13:24.984232 29634 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37399
I20260620 19:13:24.984313 29634 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:24.984453 29634 heartbeater.cc:511] Master 127.19.66.254:37399 requested a full tablet report, sending...
I20260620 19:13:24.984943 29446 ts_manager.cc:194] Registered new tserver with Master: 1801cdd35139484b807bf0cae1d0d7ac (127.19.66.193:44973)
I20260620 19:13:24.987149 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29506
I20260620 19:13:24.987218 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:24.987350 19723 external_mini_cluster.cc:1428] Setting key eedbb848987c21a47ff200ccdb25324a
I20260620 19:13:24.988088 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--builtin_ntp_servers=127.19.66.212:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 19:13:25.066166 29637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:25.066366 29637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:25.066388 29637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:25.067549 29637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:25.067589 29637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:25.067617 29637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:25.068993 29637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:25.069216 29637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:25.069482 29637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:25.074127 29642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:25.074126 29645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:25.074131 29643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:25.074368 29637 server_base.cc:1034] running on GCE node
I20260620 19:13:25.074549 29637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:25.074767 29637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:25.075902 29637 hybrid_clock.cc:648] HybridClock initialized: now 1781982805075896 us; error 33 us; skew 500 ppm
I20260620 19:13:25.077176 29637 webserver.cc:460] Webserver started at http://127.19.66.194:37445/ using document root <none> and password file <none>
I20260620 19:13:25.077414 29637 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:25.077462 29637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:25.077593 29637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:25.078625 29637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "6ef3253be137407bb40417ad63f031a1"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "71e8a23b03c9caf627132643f40ba7db"
server_key_iv: "2916ca5d20b44d7755234efe8815f4bb"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.078953 29637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "6ef3253be137407bb40417ad63f031a1"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "71e8a23b03c9caf627132643f40ba7db"
server_key_iv: "2916ca5d20b44d7755234efe8815f4bb"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.080137 29637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:25.080771 29651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.080945 29637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:25.081023 29637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "6ef3253be137407bb40417ad63f031a1"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "71e8a23b03c9caf627132643f40ba7db"
server_key_iv: "2916ca5d20b44d7755234efe8815f4bb"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.081102 29637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:25.084266 29637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:25.084496 29637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:25.084609 29637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:25.084779 29637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:25.085073 29637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:25.085105 29637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.085130 29637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:25.085147 29637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.090423 29637 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:39049
I20260620 19:13:25.090457 29764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:39049 every 8 connection(s)
I20260620 19:13:25.090739 29637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:25.092078 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29637
I20260620 19:13:25.092177 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:25.092372 19723 external_mini_cluster.cc:1428] Setting key 5bc2881129e3e0dc0d390c69de218df1
I20260620 19:13:25.093331 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--builtin_ntp_servers=127.19.66.212:38769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 19:13:25.093878 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:25.091388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:33155 (local address 127.19.66.254:37399)
0620 19:13:25.091463 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:25.091464 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:25.091644 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:13:25.092165 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:25.092167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:25.092174 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:25.092207 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:25.093676 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:25.093677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:25.093677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:25.093680 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:25.093695 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:25.093699 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:25.093748 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:25.093816 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 19:13:25.093826 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:13:25.094128 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:25.091362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37399 (local address 127.19.66.194:33155)
0620 19:13:25.091532 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:25.091570 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:25.092032 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:25.092407 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:25.092412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:25.092423 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:25.093532 (+ 1109us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:25.093533 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:25.093569 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:25.093600 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:25.093747 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:25.093750 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:25.093753 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:25.093824 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:25.093883 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260620 19:13:25.094594 29765 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37399
I20260620 19:13:25.094681 29765 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:25.094841 29765 heartbeater.cc:511] Master 127.19.66.254:37399 requested a full tablet report, sending...
I20260620 19:13:25.095156 29446 ts_manager.cc:194] Registered new tserver with Master: 6ef3253be137407bb40417ad63f031a1 (127.19.66.194:39049)
W20260620 19:13:25.170770 29768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:25.170974 29768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:25.170993 29768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:25.172144 29768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:25.172178 29768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:25.172202 29768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:25.173550 29768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29768
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:25.173763 29768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:25.174019 29768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:25.178532 29773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:25.178524 29776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:25.178618 29768 server_base.cc:1034] running on GCE node
W20260620 19:13:25.178532 29774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:25.178925 29768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:25.179157 29768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:25.180298 29768 hybrid_clock.cc:648] HybridClock initialized: now 1781982805180285 us; error 29 us; skew 500 ppm
I20260620 19:13:25.181421 29768 webserver.cc:460] Webserver started at http://127.19.66.195:33113/ using document root <none> and password file <none>
I20260620 19:13:25.181629 29768 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:25.181675 29768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:25.181793 29768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:25.182785 29768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "dec38664418a4756a00c9881735bd42b"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "46cfaf8af1cc917d3c40988d3de96c5c"
server_key_iv: "a7ecc5201816e961dca67289cde97f67"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.183110 29768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "dec38664418a4756a00c9881735bd42b"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "46cfaf8af1cc917d3c40988d3de96c5c"
server_key_iv: "a7ecc5201816e961dca67289cde97f67"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.184351 29768 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:13:25.185037 29782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.185170 29768 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:25.185248 29768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "dec38664418a4756a00c9881735bd42b"
format_stamp: "Formatted at 2026-06-20 19:13:25 on dist-test-slave-nq8r"
server_key: "46cfaf8af1cc917d3c40988d3de96c5c"
server_key_iv: "a7ecc5201816e961dca67289cde97f67"
server_key_version: "encryptionkey@0"
I20260620 19:13:25.185313 29768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:25.188386 29768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:25.188617 29768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:25.188730 29768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:25.188936 29768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:25.189205 29768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:25.189235 29768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.189260 29768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:25.189283 29768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:25.194437 29768 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:37447
I20260620 19:13:25.194469 29895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:37447 every 8 connection(s)
I20260620 19:13:25.194736 29768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:25.197302 29486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:25.195357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:37521 (local address 127.19.66.254:37399)
0620 19:13:25.195401 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:25.195403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:25.195616 (+ 213us) server_negotiation.cc:407] Connection header received
0620 19:13:25.196026 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:25.196027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:25.196031 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:25.196079 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:25.197144 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:25.197144 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:25.197145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:13:25.197148 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:25.197157 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:13:25.197160 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:25.197199 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:25.197244 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 19:13:25.197254 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:13:25.197465 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:25.195355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37399 (local address 127.19.66.195:37521)
0620 19:13:25.195512 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:25.195542 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:25.195953 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:25.196127 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:25.196130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:25.196138 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:25.197027 (+ 889us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:13:25.197027 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:25.197054 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:13:25.197081 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:25.197199 (+ 118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:25.197201 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:25.197203 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:13:25.197249 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:13:25.197292 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260620 19:13:25.197687 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29768
I20260620 19:13:25.197767 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:25.197927 29896 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37399
I20260620 19:13:25.197959 19723 external_mini_cluster.cc:1428] Setting key 6ce585a0dbe6bb57166ab2a717c34676
I20260620 19:13:25.198012 29896 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:25.198143 29896 heartbeater.cc:511] Master 127.19.66.254:37399 requested a full tablet report, sending...
I20260620 19:13:25.198455 29446 ts_manager.cc:194] Registered new tserver with Master: dec38664418a4756a00c9881735bd42b (127.19.66.195:37447)
I20260620 19:13:25.198832 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 19:13:25.200343 29486 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:13:25.199985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.19.66.254:37399)
0620 19:13:25.200035 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:25.200037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:25.200121 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:13:25.200159 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:25.200160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:25.200163 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:25.200204 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:25.200291 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58430: BlockingRecv error: recv got EOF from 127.0.0.1:58430 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:13:25.200407 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29506
I20260620 19:13:25.205201 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29637
I20260620 19:13:25.209220 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29768
I20260620 19:13:25.212539 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29415
2026-06-20T19:13:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (499 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:25 dist-test-slave-nq8r krb5kdc[29906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:25 dist-test-slave-nq8r krb5kdc[29906](info): set up 2 sockets
Jun 20 19:13:25 dist-test-slave-nq8r krb5kdc[29906](info): commencing operation
krb5kdc: starting...
W20260620 19:13:26.165375 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:13:26.269867 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43307
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:26.346935 29919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:26.347139 29919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:26.347168 29919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:26.348294 29919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:13:26.348342 29919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:26.348362 29919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:26.348377 29919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:13:26.348393 29919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:13:26.349840 29919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:26.350070 29919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:26.350260 29919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:26.354565 29925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.354617 29924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.354647 29919 server_base.cc:1034] running on GCE node
W20260620 19:13:26.354796 29927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.355017 29919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:26.355279 29919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:26.356413 29919 hybrid_clock.cc:648] HybridClock initialized: now 1781982806356400 us; error 29 us; skew 500 ppm
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:26.358659 29919 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:13:26.359046 29919 webserver.cc:460] Webserver started at http://127.19.66.254:34065/ using document root <none> and password file <none>
I20260620 19:13:26.359258 29919 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:26.359313 29919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:26.359431 29919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:26.360483 29919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "42619690b98140b0bfc8a0d357a17c36"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "2966e23eced6f747530301d9d1f60e32"
server_key_iv: "17b47660c74f29ead838dc6a82fc555a"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.360800 29919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "42619690b98140b0bfc8a0d357a17c36"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "2966e23eced6f747530301d9d1f60e32"
server_key_iv: "17b47660c74f29ead838dc6a82fc555a"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.362002 29919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:13:26.362618 29934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.362758 29919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.362843 29919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "42619690b98140b0bfc8a0d357a17c36"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "2966e23eced6f747530301d9d1f60e32"
server_key_iv: "17b47660c74f29ead838dc6a82fc555a"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.362905 29919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:26.376283 29919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:26.378201 29919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:26.378322 29919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:26.381847 29919 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:43307
I20260620 19:13:26.381883 29986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:43307 every 8 connection(s)
I20260620 19:13:26.382180 29919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:13:26.383049 29987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:26.383602 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 29919
I20260620 19:13:26.383685 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:13:26.383829 19723 external_mini_cluster.cc:1428] Setting key 034cc814e4fcdd6d79292bf3fbdc2418
I20260620 19:13:26.385744 29987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: Bootstrap starting.
I20260620 19:13:26.386385 29987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:26.386674 29987 log.cc:824] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:26.387305 29987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: No bootstrap required, opened a new log
I20260620 19:13:26.389278 29987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } }
I20260620 19:13:26.389410 29987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:26.389431 29987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42619690b98140b0bfc8a0d357a17c36, State: Initialized, Role: FOLLOWER
I20260620 19:13:26.389536 29987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } }
I20260620 19:13:26.389616 29987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:13:26.389648 29987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:13:26.389694 29987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:26.390300 29987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } }
I20260620 19:13:26.390405 29987 leader_election.cc:302] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42619690b98140b0bfc8a0d357a17c36; no voters:
I20260620 19:13:26.390545 29987 leader_election.cc:288] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:13:26.390587 29992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:26.390741 29992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [term 1 LEADER]: Becoming Leader. State: Replica: 42619690b98140b0bfc8a0d357a17c36, State: Running, Role: LEADER
I20260620 19:13:26.390787 29987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:13:26.390856 29992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } }
I20260620 19:13:26.391157 29992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42619690b98140b0bfc8a0d357a17c36. Latest consensus state: current_term: 1 leader_uuid: "42619690b98140b0bfc8a0d357a17c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } } }
I20260620 19:13:26.391389 29992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:26.391321 29994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42619690b98140b0bfc8a0d357a17c36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42619690b98140b0bfc8a0d357a17c36" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43307 } } }
I20260620 19:13:26.391777 29994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36 [sys.catalog]: This master's current role is: LEADER
I20260620 19:13:26.391965 30001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:13:26.392781 30001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:13:26.394142 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.384284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.19.66.254:43307)
0620 19:13:26.384449 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.384458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.384489 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:13:26.385034 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.385043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.385106 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.385251 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:26.385755 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.387022 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.387764 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.388375 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.388487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.390699 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:26.390703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:26.390704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:26.390715 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:26.392597 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.393039 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.393042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.393043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.393139 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.393335 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.393336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.393336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.393570 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:26.393705 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.393814 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.393996 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260620 19:13:26.394636 30001 catalog_manager.cc:1347] Generated new cluster ID: 48da86d6740c4fdd96a3ab9e451724cd
I20260620 19:13:26.394678 30001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:13:26.403424 30001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:13:26.404104 30001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:13:26.410552 30001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: Generated new TSK 0
I20260620 19:13:26.410702 30001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:13:26.437160 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:26.515645 30013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:26.515856 30013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:26.515878 30013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:26.517042 30013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:26.517081 30013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:26.517107 30013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:13:26.518538 30013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:26.518739 30013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:26.518914 30013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:26.523468 30018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.523551 30019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.523474 30013 server_base.cc:1034] running on GCE node
W20260620 19:13:26.523497 30021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.523826 30013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:26.524045 30013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:26.525190 30013 hybrid_clock.cc:648] HybridClock initialized: now 1781982806525176 us; error 30 us; skew 500 ppm
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:26.527454 30013 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:13:26.527865 30013 webserver.cc:460] Webserver started at http://127.19.66.193:33583/ using document root <none> and password file <none>
I20260620 19:13:26.528076 30013 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:26.528124 30013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:26.528247 30013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:26.529282 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "5794a29b4fec48cc995af0e352392729"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "7d6036dce7cccfc898b94863fd42d4f8"
server_key_iv: "e5f006b7c7b1c790c23162291ebaae39"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.529582 30013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "5794a29b4fec48cc995af0e352392729"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "7d6036dce7cccfc898b94863fd42d4f8"
server_key_iv: "e5f006b7c7b1c790c23162291ebaae39"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.530812 30013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:13:26.531510 30028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.531672 30013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:26.531749 30013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "5794a29b4fec48cc995af0e352392729"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "7d6036dce7cccfc898b94863fd42d4f8"
server_key_iv: "e5f006b7c7b1c790c23162291ebaae39"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.531805 30013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:26.546962 30013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:26.548712 30013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:26.548841 30013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:26.549077 30013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:26.549353 30013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:26.549384 30013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.549412 30013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:26.549429 30013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.554801 30013 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:40371
I20260620 19:13:26.554853 30141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:40371 every 8 connection(s)
I20260620 19:13:26.555160 30013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:26.561264 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 30013
I20260620 19:13:26.561355 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:13:26.561547 19723 external_mini_cluster.cc:1428] Setting key 574a1cf6cde6e5e2b2936249d768fed2
I20260620 19:13:26.563966 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.555802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:32827 (local address 127.19.66.254:43307)
0620 19:13:26.555877 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.555879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.556063 (+ 184us) server_negotiation.cc:407] Connection header received
0620 19:13:26.556546 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.556547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.556557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.556609 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:26.557553 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.558061 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.559004 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.559193 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.559242 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.561248 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:26.561249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:26.561250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:26.561257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:26.562663 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.563021 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.563022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.563023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.563079 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.563346 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.563347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.563348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.563467 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:26.563524 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.563810 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.563901 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:13:26.564168 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.555781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43307 (local address 127.19.66.193:32827)
0620 19:13:26.555947 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.555984 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.556472 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.556664 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.556667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.557270 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:26.557477 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.557482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.558128 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.558129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.558938 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.558940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.559248 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.559248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.559343 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.559879 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:26.559888 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:26.561160 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:26.562736 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.562744 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.562746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.562958 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.563132 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.563132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.563133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.563269 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.563526 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:26.563530 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:26.563687 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.563791 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.563948 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260620 19:13:26.564786 30142 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43307
I20260620 19:13:26.564911 30142 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:26.565092 30142 heartbeater.cc:511] Master 127.19.66.254:43307 requested a full tablet report, sending...
I20260620 19:13:26.565609 29951 ts_manager.cc:194] Registered new tserver with Master: 5794a29b4fec48cc995af0e352392729 (127.19.66.193:40371)
I20260620 19:13:26.566233 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:32827
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:13:26.583339 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:26.660207 30147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:26.660400 30147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:26.660421 30147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:26.661643 30147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:26.661684 30147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:26.661710 30147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:13:26.663089 30147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:26.663291 30147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:26.663498 30147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:26.668081 30155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.668093 30153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.668082 30152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.668273 30147 server_base.cc:1034] running on GCE node
I20260620 19:13:26.668579 30147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:26.668807 30147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:26.670002 30147 hybrid_clock.cc:648] HybridClock initialized: now 1781982806669986 us; error 29 us; skew 500 ppm
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:26.672351 30147 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:13:26.672761 30147 webserver.cc:460] Webserver started at http://127.19.66.194:39493/ using document root <none> and password file <none>
I20260620 19:13:26.672974 30147 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:26.673022 30147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:26.673141 30147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:26.674098 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "eac09f13dcb941da863462f9f85af45e"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "c87e9934cea9954cae2f89f241ca4ee6"
server_key_iv: "5a87fba84969017749151c0cadfb17f1"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.674419 30147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "eac09f13dcb941da863462f9f85af45e"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "c87e9934cea9954cae2f89f241ca4ee6"
server_key_iv: "5a87fba84969017749151c0cadfb17f1"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.675601 30147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:26.676183 30162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.676326 30147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:26.676402 30147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "eac09f13dcb941da863462f9f85af45e"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "c87e9934cea9954cae2f89f241ca4ee6"
server_key_iv: "5a87fba84969017749151c0cadfb17f1"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.676470 30147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:26.695478 30147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:26.697242 30147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:26.697371 30147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:26.697576 30147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:26.697860 30147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:26.697892 30147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.697921 30147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:26.697949 30147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.703125 30147 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:36987
I20260620 19:13:26.703183 30275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:36987 every 8 connection(s)
I20260620 19:13:26.703465 30147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:13:26.707290 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 30147
I20260620 19:13:26.707371 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:13:26.707549 19723 external_mini_cluster.cc:1428] Setting key e254b31ee483bf668405a3d86be064cc
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:26.712219 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.704099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:42945 (local address 127.19.66.254:43307)
0620 19:13:26.704157 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.704160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.704329 (+ 169us) server_negotiation.cc:407] Connection header received
0620 19:13:26.704839 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.704841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.704868 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.704915 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:26.705850 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.706342 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.707283 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.707536 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.707590 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.709777 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:26.709779 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:26.709779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:26.709787 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:26.710983 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.711372 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.711374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.711374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.711429 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.711641 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.711642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.711642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.711750 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:26.711803 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.712053 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.712131 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:13:26.712390 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.704074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43307 (local address 127.19.66.194:42945)
0620 19:13:26.704215 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.704251 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.704730 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.704996 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.704999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.705566 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:26.705750 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.705755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.706422 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.706423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.707202 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.707204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.707606 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.707607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.707720 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.708185 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:26.708192 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:26.709687 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:26.711054 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.711060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.711062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.711308 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.711483 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.711484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.711484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.711587 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.711801 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:26.711806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:26.711939 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.712025 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.712184 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260620 19:13:26.712925 30276 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43307
I20260620 19:13:26.713029 30276 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:26.713209 30276 heartbeater.cc:511] Master 127.19.66.254:43307 requested a full tablet report, sending...
I20260620 19:13:26.713572 29951 ts_manager.cc:194] Registered new tserver with Master: eac09f13dcb941da863462f9f85af45e (127.19.66.194:36987)
I20260620 19:13:26.713917 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:42945
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:13:26.728677 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:44341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:13:26.806003 30281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:13:26.806203 30281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:13:26.806233 30281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:13:26.807354 30281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:13:26.807397 30281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:13:26.807432 30281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:13:26.808881 30281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:43307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:13:26.809115 30281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:13:26.809314 30281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:13:26.813962 30287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.813973 30289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:13:26.813972 30286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:13:26.814109 30281 server_base.cc:1034] running on GCE node
I20260620 19:13:26.814360 30281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:13:26.814589 30281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:13:26.815735 30281 hybrid_clock.cc:648] HybridClock initialized: now 1781982806815723 us; error 23 us; skew 500 ppm
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:13:26.818063 30281 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:13:26.818486 30281 webserver.cc:460] Webserver started at http://127.19.66.195:35497/ using document root <none> and password file <none>
I20260620 19:13:26.818693 30281 fs_manager.cc:359] Metadata directory not provided
I20260620 19:13:26.818739 30281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:13:26.818859 30281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:13:26.819841 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "56e0c8cec89b40ae99a8c112a4210471"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "38c4aed9402159eb7a26bc7aa49c18d3"
server_key_iv: "0960794bb35e4f076efbb24caa0ddf5e"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.820159 30281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "56e0c8cec89b40ae99a8c112a4210471"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "38c4aed9402159eb7a26bc7aa49c18d3"
server_key_iv: "0960794bb35e4f076efbb24caa0ddf5e"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.821401 30281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:13:26.822021 30296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.822163 30281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:13:26.822259 30281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "56e0c8cec89b40ae99a8c112a4210471"
format_stamp: "Formatted at 2026-06-20 19:13:26 on dist-test-slave-nq8r"
server_key: "38c4aed9402159eb7a26bc7aa49c18d3"
server_key_iv: "0960794bb35e4f076efbb24caa0ddf5e"
server_key_version: "encryptionkey@0"
I20260620 19:13:26.822316 30281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:13:26.831070 30281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:13:26.833149 30281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:13:26.833280 30281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:13:26.833503 30281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:13:26.833778 30281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:13:26.833810 30281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.833839 30281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:13:26.833858 30281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:13:26.839591 30281 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:40615
I20260620 19:13:26.839629 30409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:40615 every 8 connection(s)
I20260620 19:13:26.839941 30281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:13:26.842656 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 30281
I20260620 19:13:26.842736 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:13:26.842890 19723 external_mini_cluster.cc:1428] Setting key 12ee84f36a0b73c1500c96508eb632f9
Jun 20 19:13:26 dist-test-slave-nq8r krb5kdc[29906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982806, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:13:26.848841 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.840606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:38719 (local address 127.19.66.254:43307)
0620 19:13:26.840667 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.840669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.840864 (+ 195us) server_negotiation.cc:407] Connection header received
0620 19:13:26.841347 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.841348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.841357 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.841419 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:26.842313 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.842850 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.844110 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.844315 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.844381 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.846431 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:26.846432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:26.846432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:26.846439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:26.847759 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.848102 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.848103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.848104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.848157 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.848353 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.848354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.848354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.848456 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:26.848501 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.848708 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.848776 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:13:26.849056 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.840573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43307 (local address 127.19.66.195:38719)
0620 19:13:26.840727 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.840769 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.841260 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.841484 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.841488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.842038 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:13:26.842242 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.842246 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.842936 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.842937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.844018 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.844021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.844389 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.844389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.844502 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.845052 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:13:26.845061 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:13:26.846341 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:13:26.847821 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.847826 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.847827 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.848045 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.848206 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:13:26.848207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:13:26.848207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:13:26.848308 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:13:26.848498 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:13:26.848502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:13:26.848612 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.848689 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.848805 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260620 19:13:26.849454 30410 heartbeater.cc:347] Connected to a master server at 127.19.66.254:43307
I20260620 19:13:26.849539 30410 heartbeater.cc:464] Registering TS with master...
I20260620 19:13:26.849681 30410 heartbeater.cc:511] Master 127.19.66.254:43307 requested a full tablet report, sending...
I20260620 19:13:26.850011 29951 ts_manager.cc:194] Registered new tserver with Master: 56e0c8cec89b40ae99a8c112a4210471 (127.19.66.195:40615)
I20260620 19:13:26.850342 29951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:38719
I20260620 19:13:26.854453 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:13:26.860728 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.855643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.19.66.254:43307)
0620 19:13:26.855690 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.855691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.855793 (+ 102us) server_negotiation.cc:407] Connection header received
0620 19:13:26.855853 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.855854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.855861 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.855902 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:13:26.856372 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.856832 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.857518 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.857692 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.857733 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.858710 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:13:26.858711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:13:26.858711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:13:26.858717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:13:26.859873 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.860119 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.860120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.860120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.860172 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:13:26.860333 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:13:26.860333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:13:26.860333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:13:26.860465 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:13:26.860538 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.860609 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.860668 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:13:26.862169 29951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:13:26.863595 29951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:13:26.869589 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40371 (local address 127.0.0.1:37360)
0620 19:13:26.865787 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.865818 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.865914 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.866193 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.866194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.866196 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:26.866261 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.866263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867221 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.867222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.868397 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.868400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869275 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.869374 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869389 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.869460 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.869513 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260620 19:13:26.869652 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.19.66.195:40615)
0620 19:13:26.865822 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.865839 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.865949 (+ 110us) server_negotiation.cc:407] Connection header received
0620 19:13:26.866027 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.866029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.866045 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.866138 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:26.866356 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867476 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.868675 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869352 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869408 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869456 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.869545 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.869578 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":79,"threads_started":1}
I20260620 19:13:26.869903 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.19.66.194:36987)
0620 19:13:26.865889 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.865893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.866080 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:13:26.866121 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.866123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.866140 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.866209 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:26.866449 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867499 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.868917 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869521 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869628 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.869811 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.869847 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260620 19:13:26.870069 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.19.66.193:40371)
0620 19:13:26.865786 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:26.865790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:26.865873 (+ 83us) server_negotiation.cc:407] Connection header received
0620 19:13:26.866028 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:26.866030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:26.866045 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:26.866112 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:26.866342 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867133 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.868475 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869193 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869268 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869324 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:26.869944 (+ 620us) server_negotiation.cc:299] Negotiation successful
0620 19:13:26.869992 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260620 19:13:26.869673 30419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40615 (local address 127.0.0.1:39776)
0620 19:13:26.865635 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.865886 (+ 251us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.865949 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.866146 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.866149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.866151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:26.866278 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.866280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867577 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.867578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.868611 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.868612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869410 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.869477 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869480 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.869531 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.869570 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":29,"thread_start_us":48,"threads_started":1}
I20260620 19:13:26.869881 30422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:26.865674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:36987 (local address 127.0.0.1:55212)
0620 19:13:26.866024 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:26.866030 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:26.866064 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:26.866287 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:26.866287 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:26.866289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:26.866317 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.866319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.867592 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.867593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.868405 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:26.868407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:26.869661 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:26.869662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:26.869727 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:26.869731 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:13:26.869782 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:13:26.869814 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260620 19:13:26.870802 30210 tablet_service.cc:1467] Processing CreateTablet for tablet b2c724797b614f1891facfec1b83514c (DEFAULT_TABLE table=test-table [id=0e386708b8974704af9c4f90f04844be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:26.870805 30344 tablet_service.cc:1467] Processing CreateTablet for tablet b2c724797b614f1891facfec1b83514c (DEFAULT_TABLE table=test-table [id=0e386708b8974704af9c4f90f04844be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:26.870980 30076 tablet_service.cc:1467] Processing CreateTablet for tablet b2c724797b614f1891facfec1b83514c (DEFAULT_TABLE table=test-table [id=0e386708b8974704af9c4f90f04844be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:13:26.871151 30344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c724797b614f1891facfec1b83514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:26.871151 30210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c724797b614f1891facfec1b83514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:26.871304 30076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2c724797b614f1891facfec1b83514c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:13:26.873839 30424 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Bootstrap starting.
I20260620 19:13:26.874241 30426 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Bootstrap starting.
I20260620 19:13:26.874437 30425 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Bootstrap starting.
I20260620 19:13:26.874549 30424 tablet_bootstrap.cc:654] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:26.875056 30425 tablet_bootstrap.cc:654] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:26.875080 30424 log.cc:824] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:26.875512 30425 log.cc:824] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:26.875808 30424 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: No bootstrap required, opened a new log
I20260620 19:13:26.875875 30424 ts_tablet_manager.cc:1397] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:26.876130 30425 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: No bootstrap required, opened a new log
I20260620 19:13:26.876189 30425 ts_tablet_manager.cc:1397] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:13:26.876284 30426 tablet_bootstrap.cc:654] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Neither blocks nor log segments found. Creating new log.
I20260620 19:13:26.876825 30426 log.cc:824] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:13:26.877701 30425 raft_consensus.cc:348] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.877828 30425 raft_consensus.cc:374] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:26.877859 30425 raft_consensus.cc:729] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56e0c8cec89b40ae99a8c112a4210471, State: Initialized, Role: FOLLOWER
I20260620 19:13:26.877803 30424 raft_consensus.cc:348] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.877925 30424 raft_consensus.cc:374] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:26.877961 30424 raft_consensus.cc:729] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5794a29b4fec48cc995af0e352392729, State: Initialized, Role: FOLLOWER
I20260620 19:13:26.877982 30425 consensus_queue.cc:260] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.878072 30424 consensus_queue.cc:260] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.878263 30425 ts_tablet_manager.cc:1428] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:26.878337 30142 heartbeater.cc:503] Master 127.19.66.254:43307 was elected leader, sending a full tablet report...
I20260620 19:13:26.878275 30410 heartbeater.cc:503] Master 127.19.66.254:43307 was elected leader, sending a full tablet report...
I20260620 19:13:26.878386 30424 ts_tablet_manager.cc:1428] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:13:26.878939 30426 tablet_bootstrap.cc:492] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: No bootstrap required, opened a new log
I20260620 19:13:26.879045 30426 ts_tablet_manager.cc:1397] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 19:13:26.880514 30426 raft_consensus.cc:348] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.880632 30426 raft_consensus.cc:374] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:13:26.880652 30426 raft_consensus.cc:729] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac09f13dcb941da863462f9f85af45e, State: Initialized, Role: FOLLOWER
I20260620 19:13:26.880728 30426 consensus_queue.cc:260] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:26.880956 30426 ts_tablet_manager.cc:1428] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:13:26.881062 30276 heartbeater.cc:503] Master 127.19.66.254:43307 was elected leader, sending a full tablet report...
W20260620 19:13:26.953969 30277 tablet.cc:2369] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:13:27.055864 30143 tablet.cc:2369] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:13:27.090463 30411 tablet.cc:2369] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:13:27.111372 30432 raft_consensus.cc:482] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:13:27.111513 30432 raft_consensus.cc:504] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:27.111860 30432 leader_election.cc:288] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5794a29b4fec48cc995af0e352392729 (127.19.66.193:40371), 56e0c8cec89b40ae99a8c112a4210471 (127.19.66.195:40615)
I20260620 19:13:27.115617 30421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.112153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:57543 (local address 127.19.66.193:40371)
0620 19:13:27.112198 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:27.112200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:27.112208 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:13:27.112236 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:27.112236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:27.112244 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:27.112304 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:27.112444 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.113220 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.114701 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.115259 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.115334 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:27.115343 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:27.115509 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 19:13:27.115546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:13:27.115617 30420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.112127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:43567 (local address 127.19.66.195:40615)
0620 19:13:27.112170 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:27.112172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:27.112276 (+ 104us) server_negotiation.cc:407] Connection header received
0620 19:13:27.112302 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:27.112302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:27.112311 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:27.112376 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:13:27.112518 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.113219 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.114701 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.115259 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.115341 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:27.115347 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:27.115509 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 19:13:27.115546 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:13:27.115644 30278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.112023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40371 (local address 127.19.66.194:57543)
0620 19:13:27.112077 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:27.112090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:27.112138 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:27.112315 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:27.112316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:27.112318 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:27.112380 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:27.112382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.113294 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.113294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:27.114615 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:27.114618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.115335 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.115336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:27.115441 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:27.115447 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 19:13:27.115498 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:13:27.115549 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":109}
I20260620 19:13:27.115649 30433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.112033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40615 (local address 127.19.66.194:43567)
0620 19:13:27.112218 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:27.112225 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:13:27.112266 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:13:27.112387 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:13:27.112387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:13:27.112389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:13:27.112456 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:27.112457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.113293 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.113294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:27.114615 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:13:27.114618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.115335 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.115336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:13:27.115440 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:27.115448 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 19:13:27.115498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:13:27.115549 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":24,"thread_start_us":104,"threads_started":1}
I20260620 19:13:27.116035 30364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c724797b614f1891facfec1b83514c" candidate_uuid: "eac09f13dcb941da863462f9f85af45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e0c8cec89b40ae99a8c112a4210471" is_pre_election: true
I20260620 19:13:27.116024 30096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c724797b614f1891facfec1b83514c" candidate_uuid: "eac09f13dcb941da863462f9f85af45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5794a29b4fec48cc995af0e352392729" is_pre_election: true
I20260620 19:13:27.116164 30096 raft_consensus.cc:2454] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac09f13dcb941da863462f9f85af45e in term 0.
I20260620 19:13:27.116168 30364 raft_consensus.cc:2454] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eac09f13dcb941da863462f9f85af45e in term 0.
I20260620 19:13:27.116382 30164 leader_election.cc:302] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e0c8cec89b40ae99a8c112a4210471, eac09f13dcb941da863462f9f85af45e; no voters:
I20260620 19:13:27.116510 30432 raft_consensus.cc:2781] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:13:27.116576 30432 raft_consensus.cc:482] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:13:27.116595 30432 raft_consensus.cc:3037] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:27.117308 30432 raft_consensus.cc:504] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:27.117430 30432 leader_election.cc:288] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [CANDIDATE]: Term 1 election: Requested vote from peers 5794a29b4fec48cc995af0e352392729 (127.19.66.193:40371), 56e0c8cec89b40ae99a8c112a4210471 (127.19.66.195:40615)
I20260620 19:13:27.117576 30364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c724797b614f1891facfec1b83514c" candidate_uuid: "eac09f13dcb941da863462f9f85af45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56e0c8cec89b40ae99a8c112a4210471"
I20260620 19:13:27.117583 30096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2c724797b614f1891facfec1b83514c" candidate_uuid: "eac09f13dcb941da863462f9f85af45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5794a29b4fec48cc995af0e352392729"
I20260620 19:13:27.117648 30364 raft_consensus.cc:3037] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:27.117653 30096 raft_consensus.cc:3037] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:13:27.118307 30364 raft_consensus.cc:2454] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac09f13dcb941da863462f9f85af45e in term 1.
I20260620 19:13:27.118372 30096 raft_consensus.cc:2454] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eac09f13dcb941da863462f9f85af45e in term 1.
I20260620 19:13:27.118462 30164 leader_election.cc:302] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56e0c8cec89b40ae99a8c112a4210471, eac09f13dcb941da863462f9f85af45e; no voters:
I20260620 19:13:27.118556 30432 raft_consensus.cc:2781] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:13:27.118701 30432 raft_consensus.cc:686] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 1 LEADER]: Becoming Leader. State: Replica: eac09f13dcb941da863462f9f85af45e, State: Running, Role: LEADER
I20260620 19:13:27.118808 30432 consensus_queue.cc:237] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } }
I20260620 19:13:27.119617 29950 catalog_manager.cc:5495] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e reported cstate change: term changed from 0 to 1, leader changed from <none> to eac09f13dcb941da863462f9f85af45e (127.19.66.194). New cstate: current_term: 1 leader_uuid: "eac09f13dcb941da863462f9f85af45e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eac09f13dcb941da863462f9f85af45e" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 36987 } health_report { overall_health: HEALTHY } } }
I20260620 19:13:27.149801 30423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.147589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55218 (local address 127.19.66.194:36987)
0620 19:13:27.147643 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:13:27.147645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:13:27.147654 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:13:27.147696 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:13:27.147696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:13:27.147715 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:13:27.147775 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:13:27.147881 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.148343 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.149107 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:13:27.149289 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:13:27.149342 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:13:27.149489 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:13:27.149591 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:13:27.149663 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:13:27.149683 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 19:13:27.149702 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:13:27.151494 30364 raft_consensus.cc:1261] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 1 FOLLOWER]: Refusing update from remote peer eac09f13dcb941da863462f9f85af45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:27.151494 30096 raft_consensus.cc:1261] T b2c724797b614f1891facfec1b83514c P 5794a29b4fec48cc995af0e352392729 [term 1 FOLLOWER]: Refusing update from remote peer eac09f13dcb941da863462f9f85af45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:13:27.151835 30434 consensus_queue.cc:1035] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5794a29b4fec48cc995af0e352392729" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:27.151979 30432 consensus_queue.cc:1035] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "56e0c8cec89b40ae99a8c112a4210471" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:13:27.155135 30439 mvcc.cc:205] Tried to move back new op lower bound from 7299001578089992192 to 7299001577959133184. Current Snapshot: MvccSnapshot[applied={T|T < 7299001578089992192}]
I20260620 19:13:27.155274 30438 mvcc.cc:205] Tried to move back new op lower bound from 7299001578089992192 to 7299001577959133184. Current Snapshot: MvccSnapshot[applied={T|T < 7299001578089992192 or (T in {7299001578089992192})}]
I20260620 19:13:27.156327 29950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:27.156411 29950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37308:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:13:27.157114 29950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42619690b98140b0bfc8a0d357a17c36: Sending DeleteTablet for 3 replicas of tablet b2c724797b614f1891facfec1b83514c
I20260620 19:13:27.157433 30076 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c724797b614f1891facfec1b83514c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:27 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:37360
I20260620 19:13:27.157533 30344 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c724797b614f1891facfec1b83514c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:27 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39776
I20260620 19:13:27.157661 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 30013
I20260620 19:13:27.157758 30447 tablet_replica.cc:330] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: stopping tablet replica
I20260620 19:13:27.157874 30447 raft_consensus.cc:2229] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:13:27.157968 30447 raft_consensus.cc:2258] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:27.158213 30210 tablet_service.cc:1514] Processing DeleteTablet for tablet b2c724797b614f1891facfec1b83514c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:13:27 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55212
I20260620 19:13:27.158408 30448 tablet_replica.cc:330] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: stopping tablet replica
I20260620 19:13:27.158497 30448 raft_consensus.cc:2229] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:13:27.158586 30448 raft_consensus.cc:2258] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:13:27.158941 30448 ts_tablet_manager.cc:1905] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:27.160168 30448 ts_tablet_manager.cc:1918] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:27.160229 30448 log.cc:1196] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/b2c724797b614f1891facfec1b83514c
I20260620 19:13:27.160472 30448 ts_tablet_manager.cc:1939] T b2c724797b614f1891facfec1b83514c P eac09f13dcb941da863462f9f85af45e: Deleting consensus metadata
I20260620 19:13:27.160871 29936 catalog_manager.cc:4841] TS eac09f13dcb941da863462f9f85af45e (127.19.66.194:36987): tablet b2c724797b614f1891facfec1b83514c (table test-table [id=0e386708b8974704af9c4f90f04844be]) successfully deleted
I20260620 19:13:27.161703 30447 ts_tablet_manager.cc:1905] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:13:27.162830 30447 ts_tablet_manager.cc:1918] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:13:27.162894 30447 log.cc:1196] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/b2c724797b614f1891facfec1b83514c
I20260620 19:13:27.163151 30447 ts_tablet_manager.cc:1939] T b2c724797b614f1891facfec1b83514c P 56e0c8cec89b40ae99a8c112a4210471: Deleting consensus metadata
I20260620 19:13:27.163483 29936 catalog_manager.cc:4841] TS 56e0c8cec89b40ae99a8c112a4210471 (127.19.66.195:40615): tablet b2c724797b614f1891facfec1b83514c (table test-table [id=0e386708b8974704af9c4f90f04844be]) successfully deleted
W20260620 19:13:27.165328 29938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:40371 (error 108)
I20260620 19:13:27.165495 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 30147
I20260620 19:13:27.165717 30418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:13:27.165624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40371 (local address 127.0.0.1:37364)
0620 19:13:27.165663 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:13:27.165689 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:40371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:13:27.165799 29938 catalog_manager.cc:4584] TS 5794a29b4fec48cc995af0e352392729 (127.19.66.193:40371): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2c724797b614f1891facfec1b83514c: Network error: Client connection negotiation failed: client connection to 127.19.66.193:40371: connect: Connection refused (error 111)
I20260620 19:13:27.170082 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 30281
I20260620 19:13:27.173626 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 29919
2026-06-20T19:13:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1961 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:13:27 dist-test-slave-nq8r krb5kdc[30452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:13:27 dist-test-slave-nq8r krb5kdc[30452](info): set up 2 sockets
Jun 20 19:13:27 dist-test-slave-nq8r krb5kdc[30452](info): commencing operation
krb5kdc: starting...
W20260620 19:13:28.121551 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:13:28 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:13:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangeradmin_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangeradmin_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangerlookup_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangerlookup_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
I20260620 19:13:28.262904 19723 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiAqWb6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/postgres -l logfile start
2026-06-20 19:13:30.355 UTC [30482] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:13:30.355 UTC [30482] LOG: listening on IPv4 address "127.19.66.212", port 37959
2026-06-20 19:13:30.358 UTC [30482] LOG: listening on Unix socket "/tmp/.s.PGSQL.37959"
2026-06-20 19:13:30.384 UTC [30485] LOG: database system was shut down at 2026-06-20 19:13:29 UTC
2026-06-20 19:13:30.388 UTC [30482] LOG: database system is ready to accept connections
I20260620 19:13:31.245309 19723 mini_postgres.cc:96] Postgres bound to 37959
2026-06-20 19:13:31.250 UTC [30493] FATAL: database "slave" does not exist
127.19.66.212:37959 - accepting connections
I20260620 19:13:31.251333 19723 mini_ranger.cc:161] Starting Ranger...
I20260620 19:13:31.264559 19723 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:13:31.624244 19723 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:13:31.624318 19723 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin
2026-06-20 19:13:31,890 [I] DB FLAVOR :POSTGRES
2026-06-20 19:13:31,891 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:13:31,891 [I] Checking connection..
2026-06-20 19:13:31,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:13:32,140 [I] Checking connection passed.
2026-06-20 19:13:32,141 [I] --------- Verifying version history table ---------
2026-06-20 19:13:32,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:13:32,361 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:13:32,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:13:32,600 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:13:32,600 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:13:32,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:13:32.844 UTC [30557] WARNING: there is no transaction in progress
2026-06-20 19:13:32,849 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:13:32,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:13:33,094 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:13:33,094 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:13:33,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:13:33,326 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:13:33,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nq8r.c.gcp-upstream.internal','N') ;"
2026-06-20 19:13:33,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:13:34,045 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:13:34,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:13:34,293 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:13:34,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:13:34,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:13:34,753 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:13:34,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:13:35.629 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.644 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.657 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.670 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.685 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.700 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.748 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.754 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.766 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.775 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.784 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.796 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.804 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.810 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.815 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:35.820 UTC [30665] WARNING: there is no transaction in progress
2026-06-20 19:13:36,324 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:13:36,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:13:36,548 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:13:36,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:13:36,787 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:13:36,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:13:37,022 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:13:37,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:13:37,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:13:37,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nq8r.c.gcp-upstream.internal';"
2026-06-20 19:13:37,737 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:13:37,737 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:13:37,977 [I] DB_PATCHES have already been applied
I20260620 19:13:37.983673 19723 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiAqWb6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:13:37.983747 19723 mini_ranger.cc:193] Using host: 127.19.66.212
I20260620 19:13:37.987732 19723 mini_ranger.cc:227] Ranger admin URL: http://127.19.66.212:41747
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:13:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:13:38.995254 19723 mini_ranger.cc:160] Time spent starting Ranger: real 7.744s user 0.001s sys 0.008s
Jun 20, 2026 7:13:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:13:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:13:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:13:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:13:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:13:59.116361 19723 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangerkms_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/rangerkms_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.66.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.66.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:13:59 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982839, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:13:59.186398 19723 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:13:59.194573 19723 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:13:59.888713 19723 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:13:59.888780 19723 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms
I20260620 19:13:59.888797 19723 mini_ranger_kms.cc:227] Using postgres at 127.19.66.212:37959
2026-06-20 19:14:00,039 [I] DB FLAVOR :POSTGRES
2026-06-20 19:14:00,039 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:14:00,039 [I] Checking connection
2026-06-20 19:14:00,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:14:00,270 [I] connection success
2026-06-20 19:14:00,270 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:14:00,270 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:14:00,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:14:00,525 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:14:00,525 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:14:00,525 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:14:00,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:37959/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:14:00,767 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:14:01.238119 19723 mini_ranger_kms.cc:310] Created kms service
I20260620 19:14:01.447719 19723 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:14:01.489250 19723 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:14:01 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982839, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:01.836359 19723 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:14:01.836469 19723 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:14:01.836488 19723 mini_ranger_kms.cc:240] Using host: 127.19.66.212
I20260620 19:14:01.839895 19723 mini_ranger_kms.cc:276] Ranger KMS PID: 30845
I20260620 19:14:01.839983 19723 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.66.212:43553
Jun 20, 2026 7:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:14:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:14:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:14:03.842587 19723 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.656s user 0.005s sys 0.126s
I20260620 19:14:03.842676 19723 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:14:03.842756 19723 mini_ranger_kms.cc:401] 127.19.66.212:43553/kms/v1/keys
Jun 20 19:14:04 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982844, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:14:04 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982844, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
Jun 20 19:14:04 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982844, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
Jun 20 19:14:06 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:14:06.896755 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46535
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46535
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:43553/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:06.975600 30896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:06.975867 30896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:06.975894 30896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:06.977548 30896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:06.977602 30896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:06.977618 30896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:06.977633 30896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:06.977648 30896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:06.979893 30896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.66.212:43553/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46535
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:06.980154 30896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:06.980414 30896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:06.985621 30904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:06.985634 30896 server_base.cc:1034] running on GCE node
W20260620 19:14:06.985759 30902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:06.985889 30901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:06.986143 30896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:06.986477 30896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:06.987651 30896 hybrid_clock.cc:648] HybridClock initialized: now 1781982846987630 us; error 35 us; skew 500 ppm
Jun 20 19:14:06 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:06.990554 30896 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:06.991070 30896 webserver.cc:460] Webserver started at http://127.19.66.254:37865/ using document root <none> and password file <none>
I20260620 19:14:06.991331 30896 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:06.991384 30896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:06.991513 30896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:06 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:07.042898 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "4b22195366264b649cad0a5129e9ff09"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-nq8r"
server_key: "ec750c647b30ec5e931cbf3f08f9c60b"
server_key_iv: "05ba781e6384954335ad29e85181ac5c"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:07.043402 30896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "4b22195366264b649cad0a5129e9ff09"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-nq8r"
server_key: "ec750c647b30ec5e931cbf3f08f9c60b"
server_key_iv: "05ba781e6384954335ad29e85181ac5c"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:07.045236 30896 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 19:14:07.058868 30911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:07.059108 30896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:07.059214 30896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "4b22195366264b649cad0a5129e9ff09"
format_stamp: "Formatted at 2026-06-20 19:14:07 on dist-test-slave-nq8r"
server_key: "ec750c647b30ec5e931cbf3f08f9c60b"
server_key_iv: "05ba781e6384954335ad29e85181ac5c"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:07.059314 30896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:07.081961 30896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:07.084233 30896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:07.084388 30896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:07.089385 30896 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:46535
I20260620 19:14:07.089421 30963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:46535 every 8 connection(s)
I20260620 19:14:07.089844 30896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:07.090553 30964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:07.092151 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 30896
I20260620 19:14:07.092253 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:14:07.092450 19723 external_mini_cluster.cc:1428] Setting key c65f264e511ac674b936951522d3ec21
I20260620 19:14:07.093312 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: Bootstrap starting.
I20260620 19:14:07.094045 30964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:07.094362 30964 log.cc:824] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:07.095064 30964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: No bootstrap required, opened a new log
I20260620 19:14:07.097304 30964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } }
I20260620 19:14:07.097440 30964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:07.097470 30964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b22195366264b649cad0a5129e9ff09, State: Initialized, Role: FOLLOWER
I20260620 19:14:07.097605 30964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } }
I20260620 19:14:07.097680 30964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:07.097715 30964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:07.097754 30964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:07.098253 30964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } }
I20260620 19:14:07.098343 30964 leader_election.cc:302] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b22195366264b649cad0a5129e9ff09; no voters:
I20260620 19:14:07.098485 30964 leader_election.cc:288] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:07.098754 30964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:07.099174 30969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:07.099304 30969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [term 1 LEADER]: Becoming Leader. State: Replica: 4b22195366264b649cad0a5129e9ff09, State: Running, Role: LEADER
I20260620 19:14:07.099385 30969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } }
I20260620 19:14:07.099697 30969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b22195366264b649cad0a5129e9ff09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } } }
I20260620 19:14:07.099762 30969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:07.099862 30969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b22195366264b649cad0a5129e9ff09. Latest consensus state: current_term: 1 leader_uuid: "4b22195366264b649cad0a5129e9ff09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b22195366264b649cad0a5129e9ff09" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46535 } } }
I20260620 19:14:07.099900 30969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:07.100129 30964 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 19:14:07 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:07.103895 30981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:07.104455 30981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:07.106348 30967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:07.093193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37542 (local address 127.19.66.254:46535)
0620 19:14:07.093674 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:07.093685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:07.093726 (+ 41us) server_negotiation.cc:407] Connection header received
0620 19:14:07.094327 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:07.094337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:07.094404 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:07.094564 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:07.095286 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.096652 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.097794 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:07.098418 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:07.098540 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:07.101810 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:07.101840 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:07.101841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:07.101891 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:07.104302 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.104699 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.104702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.104702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.104797 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:07.105047 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:07.105048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:07.105048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:07.105363 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:07.105913 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:07.105984 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:14:07.106134 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":68,"threads_started":1}
I20260620 19:14:07.112783 30981 catalog_manager.cc:1347] Generated new cluster ID: de0e4087483c414ca92952de00abdd30
I20260620 19:14:07.112879 30981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:07.134016 30981 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:07.134754 30981 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:07.142190 30981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: Generated new TSK 0
I20260620 19:14:07.142390 30981 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:14:07.906725 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:43553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:07.989776 31009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:07.989977 31009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:07.990006 31009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:07.991223 31009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:07.991264 31009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:07.991299 31009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:14:07.992834 31009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.66.212:43553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:07.993094 31009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:07.993310 31009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:07.998054 31014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:07.998067 31017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:07.998018 31015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:07.998411 31009 server_base.cc:1034] running on GCE node
I20260620 19:14:07.998617 31009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:07.998857 31009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:08.002074 31009 hybrid_clock.cc:648] HybridClock initialized: now 1781982848002057 us; error 34 us; skew 500 ppm
Jun 20 19:14:07 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:08.004688 31009 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:14:08.005285 31009 webserver.cc:460] Webserver started at http://127.19.66.193:43729/ using document root <none> and password file <none>
I20260620 19:14:08.005560 31009 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:08.005635 31009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:08.005779 31009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:08.020067 31009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "4d2d4bd7b4b2593da00cb356bd557aa5"
server_key_iv: "0c6c32db1d0c1398181d3466eb962dea"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.020551 31009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "4d2d4bd7b4b2593da00cb356bd557aa5"
server_key_iv: "0c6c32db1d0c1398181d3466eb962dea"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.026101 31009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260620 19:14:08.041630 31024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.041865 31009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:08.041955 31009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "4d2d4bd7b4b2593da00cb356bd557aa5"
server_key_iv: "0c6c32db1d0c1398181d3466eb962dea"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.042043 31009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:08.052548 31009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:08.054978 31009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:08.055138 31009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:08.055379 31009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:08.055763 31009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:08.055814 31009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.055845 31009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:08.055900 31009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.062069 31009 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:35343
I20260620 19:14:08.062350 31138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:35343 every 8 connection(s)
I20260620 19:14:08.062578 31009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:14:08.064803 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 31009
I20260620 19:14:08.064929 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:14:08.065127 19723 external_mini_cluster.cc:1428] Setting key 670761fd9e9873178a26997c977f508f
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982847, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:08.091383 31141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.063350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46535 (local address 127.19.66.193:52611)
0620 19:14:08.073010 (+ 9660us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.073065 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.073829 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.074165 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.074170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.079872 (+ 5702us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:08.080200 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.080211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.081988 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.081989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.083143 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.083148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.083668 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.083669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.083841 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.084565 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:08.084576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:08.087842 (+ 3266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:08.089802 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.089811 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.089813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.090108 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.090322 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.090322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.090323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.090451 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.090727 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:08.090733 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:08.090905 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.090992 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.091152 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9603,"thread_start_us":76,"threads_started":1}
I20260620 19:14:08.092130 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.063377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:52611 (local address 127.19.66.254:46535)
0620 19:14:08.063515 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.063517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.073175 (+ 9658us) server_negotiation.cc:407] Connection header received
0620 19:14:08.073948 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.073950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.073966 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.074038 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:08.080362 (+ 6324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.081800 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.083260 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.083562 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.085332 (+ 1770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.087944 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:08.087946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:08.087946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:08.087954 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:08.089649 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.090192 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.090192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.090193 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.090255 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.090512 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.090513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.090513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.090644 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:08.091922 (+ 1278us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.091964 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.092048 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260620 19:14:08.092680 31139 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46535
I20260620 19:14:08.092796 31139 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:08.093004 31139 heartbeater.cc:511] Master 127.19.66.254:46535 requested a full tablet report, sending...
I20260620 19:14:08.093550 30928 ts_manager.cc:194] Registered new tserver with Master: 8a32ee3dcbb14727b8af6a8b73dbcc14 (127.19.66.193:35343)
I20260620 19:14:08.094305 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:52611
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:14:08.098390 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:43553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:08.208292 31146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:08.208562 31146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:08.208600 31146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:08.210301 31146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:08.210359 31146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:08.210395 31146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:14:08.212504 31146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.66.212:43553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:08.212747 31146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:08.213068 31146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:08.219050 31156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:08.219153 31153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:08.219049 31154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:08.219951 31146 server_base.cc:1034] running on GCE node
I20260620 19:14:08.220156 31146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:08.220444 31146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:08.221619 31146 hybrid_clock.cc:648] HybridClock initialized: now 1781982848221598 us; error 34 us; skew 500 ppm
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:08.224713 31146 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:14:08.225307 31146 webserver.cc:460] Webserver started at http://127.19.66.194:43639/ using document root <none> and password file <none>
I20260620 19:14:08.225584 31146 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:08.225647 31146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:08.225852 31146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:08.240064 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "343aed55273742a3b112be9f5014e5da"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "8a1e76cc19e021e337ad97572fcd8c4c"
server_key_iv: "f66313345b8f70a88579de16ecbde1da"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.240540 31146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "343aed55273742a3b112be9f5014e5da"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "8a1e76cc19e021e337ad97572fcd8c4c"
server_key_iv: "f66313345b8f70a88579de16ecbde1da"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.243160 31146 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:08.267030 31163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.267285 31146 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260620 19:14:08.267382 31146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "343aed55273742a3b112be9f5014e5da"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "8a1e76cc19e021e337ad97572fcd8c4c"
server_key_iv: "f66313345b8f70a88579de16ecbde1da"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.267489 31146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:08.285257 31146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:08.294420 31146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:08.294627 31146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:08.294883 31146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:08.295300 31146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:08.298658 31146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 19:14:08.298779 31146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:08.298893 31146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.308099 31146 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:34029
I20260620 19:14:08.308586 31146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:08.309894 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 31146
I20260620 19:14:08.309990 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:14:08.310160 19723 external_mini_cluster.cc:1428] Setting key a0345ce633ca0bc91d87bd7d05e7a666
I20260620 19:14:08.313311 31276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:34029 every 8 connection(s)
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:08.325955 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.314057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:33673 (local address 127.19.66.254:46535)
0620 19:14:08.314101 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.314103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.314343 (+ 240us) server_negotiation.cc:407] Connection header received
0620 19:14:08.315004 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.315006 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.315022 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.315052 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:08.315765 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.316533 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.318092 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.318410 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.318454 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.322745 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:08.322747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:08.322747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:08.322755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:08.324252 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.324795 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.324796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.324796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.324883 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.325186 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.325187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.325187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.325327 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:08.325392 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.325755 (+ 363us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.325854 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:14:08.326134 31280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.314063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46535 (local address 127.19.66.194:33673)
0620 19:14:08.314221 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.314266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.314887 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.315236 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.315240 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.315463 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:08.315663 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.315667 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.316650 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.316651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.317989 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.317992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.318552 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.318553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.318699 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.319298 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:08.319310 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:08.322639 (+ 3329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:08.324358 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.324368 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.324370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.324703 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.324950 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.324951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.324951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.325112 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.325398 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:08.325405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:08.325584 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.325715 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.325889 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260620 19:14:08.326762 31277 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46535
I20260620 19:14:08.326867 31277 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:08.327075 31277 heartbeater.cc:511] Master 127.19.66.254:46535 requested a full tablet report, sending...
I20260620 19:14:08.327502 30928 ts_manager.cc:194] Registered new tserver with Master: 343aed55273742a3b112be9f5014e5da (127.19.66.194:34029)
I20260620 19:14:08.327996 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:33673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:14:08.336450 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:43553/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:08.422801 31283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:08.423146 31283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:08.423239 31283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:08.425122 31283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:08.425240 31283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:08.425293 31283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:14:08.427115 31283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.66.212:43553/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:46535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:08.427357 31283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:08.427577 31283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:08.434253 31289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:08.434649 31288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:08.434919 31291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:08.435591 31283 server_base.cc:1034] running on GCE node
I20260620 19:14:08.435807 31283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:08.436043 31283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:08.437177 31283 hybrid_clock.cc:648] HybridClock initialized: now 1781982848437163 us; error 29 us; skew 500 ppm
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:08.439531 31283 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:14:08.439908 31283 webserver.cc:460] Webserver started at http://127.19.66.195:33603/ using document root <none> and password file <none>
I20260620 19:14:08.440171 31283 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:08.440223 31283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:08.440366 31283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:08.455857 31283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "09bd1a231e7d4514ab5d8d261fe857dc"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "a74d05bf7c94b7bd0a7d24a5617a5e3b"
server_key_iv: "a3839717d7f076c719e28d3938094241"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.456300 31283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "09bd1a231e7d4514ab5d8d261fe857dc"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "a74d05bf7c94b7bd0a7d24a5617a5e3b"
server_key_iv: "a3839717d7f076c719e28d3938094241"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.461649 31283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 19:14:08.481021 31298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.481299 31283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 19:14:08.481411 31283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "09bd1a231e7d4514ab5d8d261fe857dc"
format_stamp: "Formatted at 2026-06-20 19:14:08 on dist-test-slave-nq8r"
server_key: "a74d05bf7c94b7bd0a7d24a5617a5e3b"
server_key_iv: "a3839717d7f076c719e28d3938094241"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:08.481515 31283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:08.499859 31283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:08.502128 31283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:08.502345 31283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:08.502561 31283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:08.502915 31283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:08.502952 31283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.502980 31283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:08.503022 31283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:08.509387 31283 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33021
I20260620 19:14:08.509865 31283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:14:08.510970 31411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33021 every 8 connection(s)
I20260620 19:14:08.512367 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 31283
I20260620 19:14:08.512458 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:14:08.512626 19723 external_mini_cluster.cc:1428] Setting key 8d672f9556be9d9720570e8f4b507411
Jun 20 19:14:08 dist-test-slave-nq8r krb5kdc[30452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982848, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:08.519347 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.511412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:45033 (local address 127.19.66.254:46535)
0620 19:14:08.511461 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.511463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.511474 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:14:08.511961 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.511962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.511975 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.512025 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:08.512975 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.513481 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.514661 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.514867 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.514920 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.516880 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:08.516882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:08.516882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:08.516889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:08.518102 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.518482 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.518483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.518484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.518536 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.518760 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.518761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.518761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.518866 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:08.518919 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.519175 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.519260 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:14:08.519505 31414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.511117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:46535 (local address 127.19.66.195:45033)
0620 19:14:08.511305 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.511351 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.511860 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.512100 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.512104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.512380 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:08.512667 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.512673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.513628 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.513629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.514582 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.514584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.514935 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.514936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.515063 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.515557 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:08.515566 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:08.516734 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:08.518172 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.518179 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.518181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.518392 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.518587 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:08.518588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:08.518589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:08.518698 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:08.518917 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:08.518923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:08.519061 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.519148 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.519286 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260620 19:14:08.519925 31412 heartbeater.cc:347] Connected to a master server at 127.19.66.254:46535
I20260620 19:14:08.520020 31412 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:08.520191 31412 heartbeater.cc:511] Master 127.19.66.254:46535 requested a full tablet report, sending...
I20260620 19:14:08.521067 30928 ts_manager.cc:194] Registered new tserver with Master: 09bd1a231e7d4514ab5d8d261fe857dc (127.19.66.195:33021)
I20260620 19:14:08.521494 30928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:45033
I20260620 19:14:08.524705 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:08.532213 31142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.526232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.19.66.254:46535)
0620 19:14:08.526276 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.526278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.526390 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:14:08.526432 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.526433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.526442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.526495 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:08.526998 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.527668 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.528379 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.528645 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.528724 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.529913 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:08.529915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:08.529915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:08.529923 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:08.531267 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.531562 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.531563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.531564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.531622 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:08.531769 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:08.531769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:08.531769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:08.531883 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:08.531972 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.532056 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.532127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:14:08.533998 30928 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:08.536248 30928 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:08.543671 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.538670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33021 (local address 127.0.0.1:33262)
0620 19:14:08.539204 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.539249 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.539325 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.539582 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.539585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.539588 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:08.539682 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.539684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.541060 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.541061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.542323 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.542327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.543370 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.543371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.543458 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.543478 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.543528 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.543584 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":87,"threads_started":1}
I20260620 19:14:08.544224 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.538670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:34029 (local address 127.0.0.1:34416)
0620 19:14:08.539532 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.539541 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.539583 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.540534 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.540535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.540537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:08.540596 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.540597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.541938 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.541938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.543080 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.543083 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.544002 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.544003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.544082 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.544086 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.544116 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.544154 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"mutex_wait_us":27,"thread_start_us":73,"threads_started":1}
I20260620 19:14:08.544401 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.539001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.19.66.195:33021)
0620 19:14:08.539167 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.539172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.539312 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:14:08.539422 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.539424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.539440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.539770 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:08.539859 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.540959 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.542457 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.543265 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.544145 (+ 880us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.544243 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.544273 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.544319 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 19:14:08.544600 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.538875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:35343 (local address 127.0.0.1:55648)
0620 19:14:08.540220 (+ 1345us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.540229 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.540275 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.540572 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.540573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.540574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:08.540600 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.540601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.541861 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.541862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.543200 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.543202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.544130 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.544131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.544179 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.544183 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.544488 (+ 305us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.544528 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":62,"threads_started":1}
I20260620 19:14:08.544963 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.538967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.19.66.193:35343)
0620 19:14:08.539482 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.539487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.540348 (+ 861us) server_negotiation.cc:407] Connection header received
0620 19:14:08.540391 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.540393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.540409 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.540470 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:08.540667 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.541775 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.543289 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.543967 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.544020 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.544101 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.544823 (+ 722us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.544885 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":69,"threads_started":1}
I20260620 19:14:08.545115 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.538856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.19.66.194:34029)
0620 19:14:08.540386 (+ 1530us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.540391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.540399 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:08.540434 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.540435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.540449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.540662 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:08.540726 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.541864 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.543180 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.543905 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.544653 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.544902 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.544925 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.544965 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":1268,"threads_started":1}
I20260620 19:14:08.545238 31345 tablet_service.cc:1467] Processing CreateTablet for tablet a80dbe1943254cfca1b514baa181b786 (DEFAULT_TABLE table=test-table [id=7b623a6033294e5eab2ee16589b25c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:08.545647 31345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80dbe1943254cfca1b514baa181b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:08.545617 31211 tablet_service.cc:1467] Processing CreateTablet for tablet a80dbe1943254cfca1b514baa181b786 (DEFAULT_TABLE table=test-table [id=7b623a6033294e5eab2ee16589b25c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:08.545895 31211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80dbe1943254cfca1b514baa181b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:08.548568 31430 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: Bootstrap starting.
I20260620 19:14:08.549495 31054 tablet_service.cc:1467] Processing CreateTablet for tablet a80dbe1943254cfca1b514baa181b786 (DEFAULT_TABLE table=test-table [id=7b623a6033294e5eab2ee16589b25c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:08.549796 31054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a80dbe1943254cfca1b514baa181b786. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:08.549104 31431 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Bootstrap starting.
I20260620 19:14:08.561717 31430 tablet_bootstrap.cc:654] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:08.562096 31430 log.cc:824] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:08.565340 31431 tablet_bootstrap.cc:654] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:08.565737 31431 log.cc:824] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:08.566141 31432 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: Bootstrap starting.
I20260620 19:14:08.566732 31431 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: No bootstrap required, opened a new log
I20260620 19:14:08.566864 31431 ts_tablet_manager.cc:1397] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.002s
I20260620 19:14:08.567134 31432 tablet_bootstrap.cc:654] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:08.567441 31432 log.cc:824] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:08.568308 31430 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: No bootstrap required, opened a new log
I20260620 19:14:08.568388 31430 ts_tablet_manager.cc:1397] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.002s
I20260620 19:14:08.568432 31432 tablet_bootstrap.cc:492] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: No bootstrap required, opened a new log
I20260620 19:14:08.568495 31432 ts_tablet_manager.cc:1397] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:08.569401 31431 raft_consensus.cc:348] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.569689 31431 raft_consensus.cc:374] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:08.569772 31431 raft_consensus.cc:729] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bd1a231e7d4514ab5d8d261fe857dc, State: Initialized, Role: FOLLOWER
I20260620 19:14:08.569988 31431 consensus_queue.cc:260] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.570408 31430 raft_consensus.cc:348] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.570407 31432 raft_consensus.cc:348] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.570566 31432 raft_consensus.cc:374] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:08.570566 31430 raft_consensus.cc:374] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:08.570598 31432 raft_consensus.cc:729] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a32ee3dcbb14727b8af6a8b73dbcc14, State: Initialized, Role: FOLLOWER
I20260620 19:14:08.570598 31430 raft_consensus.cc:729] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 343aed55273742a3b112be9f5014e5da, State: Initialized, Role: FOLLOWER
I20260620 19:14:08.570721 31432 consensus_queue.cc:260] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.570732 31430 consensus_queue.cc:260] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.570973 31430 ts_tablet_manager.cc:1428] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:14:08.570973 31432 ts_tablet_manager.cc:1428] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:08.570977 31431 ts_tablet_manager.cc:1428] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260620 19:14:08.571091 31277 heartbeater.cc:503] Master 127.19.66.254:46535 was elected leader, sending a full tablet report...
I20260620 19:14:08.571091 31139 heartbeater.cc:503] Master 127.19.66.254:46535 was elected leader, sending a full tablet report...
I20260620 19:14:08.571218 31412 heartbeater.cc:503] Master 127.19.66.254:46535 was elected leader, sending a full tablet report...
I20260620 19:14:08.605041 31438 raft_consensus.cc:482] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:08.605188 31438 raft_consensus.cc:504] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.605571 31438 leader_election.cc:288] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09bd1a231e7d4514ab5d8d261fe857dc (127.19.66.195:33021), 343aed55273742a3b112be9f5014e5da (127.19.66.194:34029)
I20260620 19:14:08.610296 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.606011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:60855 (local address 127.19.66.194:34029)
0620 19:14:08.606054 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.606055 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.606195 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:14:08.606302 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.606303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.606317 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.606345 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:08.606578 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.607349 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.609340 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.609881 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.609943 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.609953 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.610176 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.610219 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:14:08.610735 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.605792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:34029 (local address 127.19.66.193:60855)
0620 19:14:08.606127 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.606136 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.606226 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.606410 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.606411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.606412 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:08.606495 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.606497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.607848 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.607849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.609224 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.609226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.610011 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.610012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.610097 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.610102 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.610599 (+ 497us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.610648 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":32,"thread_start_us":97,"threads_started":1}
I20260620 19:14:08.611231 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80dbe1943254cfca1b514baa181b786" candidate_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343aed55273742a3b112be9f5014e5da" is_pre_election: true
I20260620 19:14:08.611405 31231 raft_consensus.cc:2454] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a32ee3dcbb14727b8af6a8b73dbcc14 in term 0.
I20260620 19:14:08.611649 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.605984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:50247 (local address 127.19.66.195:33021)
0620 19:14:08.606047 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.606049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.606203 (+ 154us) server_negotiation.cc:407] Connection header received
0620 19:14:08.606304 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.606305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.606317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.606343 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:08.607012 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.607750 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.610224 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.610798 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.610842 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.610849 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.611519 (+ 670us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.611582 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 19:14:08.611752 31028 leader_election.cc:302] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343aed55273742a3b112be9f5014e5da, 8a32ee3dcbb14727b8af6a8b73dbcc14; no voters:
I20260620 19:14:08.611896 31438 raft_consensus.cc:2781] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:08.611938 31438 raft_consensus.cc:482] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:08.612979 31438 raft_consensus.cc:3037] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:08.613886 31438 raft_consensus.cc:504] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.614053 31438 leader_election.cc:288] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [CANDIDATE]: Term 1 election: Requested vote from peers 09bd1a231e7d4514ab5d8d261fe857dc (127.19.66.195:33021), 343aed55273742a3b112be9f5014e5da (127.19.66.194:34029)
I20260620 19:14:08.611919 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.605763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33021 (local address 127.19.66.193:50247)
0620 19:14:08.606103 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.606112 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:08.606232 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:08.606406 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:08.606407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:08.606409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:08.606510 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.606512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.607855 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.607856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.610121 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:08.610124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.611361 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.611362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:08.611437 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.611441 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:08.611792 (+ 351us) client_negotiation.cc:240] Negotiation successful
0620 19:14:08.611841 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":92,"thread_start_us":55,"threads_started":1}
I20260620 19:14:08.614279 31231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80dbe1943254cfca1b514baa181b786" candidate_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "343aed55273742a3b112be9f5014e5da"
I20260620 19:14:08.614358 31231 raft_consensus.cc:3037] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:08.614704 31347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80dbe1943254cfca1b514baa181b786" candidate_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" is_pre_election: true
I20260620 19:14:08.614846 31347 raft_consensus.cc:2454] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a32ee3dcbb14727b8af6a8b73dbcc14 in term 0.
I20260620 19:14:08.615178 31231 raft_consensus.cc:2454] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a32ee3dcbb14727b8af6a8b73dbcc14 in term 1.
I20260620 19:14:08.615172 31352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a80dbe1943254cfca1b514baa181b786" candidate_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09bd1a231e7d4514ab5d8d261fe857dc"
I20260620 19:14:08.615243 31352 raft_consensus.cc:3037] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:08.615343 31028 leader_election.cc:302] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 343aed55273742a3b112be9f5014e5da, 8a32ee3dcbb14727b8af6a8b73dbcc14; no voters:
I20260620 19:14:08.615413 31438 raft_consensus.cc:2781] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:08.615542 31438 raft_consensus.cc:686] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [term 1 LEADER]: Becoming Leader. State: Replica: 8a32ee3dcbb14727b8af6a8b73dbcc14, State: Running, Role: LEADER
I20260620 19:14:08.615624 31438 consensus_queue.cc:237] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } }
I20260620 19:14:08.616019 31352 raft_consensus.cc:2454] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a32ee3dcbb14727b8af6a8b73dbcc14 in term 1.
I20260620 19:14:08.616474 30928 catalog_manager.cc:5495] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a32ee3dcbb14727b8af6a8b73dbcc14 (127.19.66.193). New cstate: current_term: 1 leader_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a32ee3dcbb14727b8af6a8b73dbcc14" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 35343 } health_report { overall_health: HEALTHY } } }
I20260620 19:14:08.627862 31427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.624537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.19.66.193:35343)
0620 19:14:08.624577 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:08.624579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:08.624588 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:14:08.624677 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:08.624678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:08.624700 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:08.624755 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:08.624950 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.625838 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.626972 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:08.627281 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:08.627352 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:08.627516 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:08.627640 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:08.627726 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:08.627749 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 19:14:08.627770 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:14:08.630009 31352 raft_consensus.cc:1261] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 1 FOLLOWER]: Refusing update from remote peer 8a32ee3dcbb14727b8af6a8b73dbcc14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:08.630050 31231 raft_consensus.cc:1261] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 1 FOLLOWER]: Refusing update from remote peer 8a32ee3dcbb14727b8af6a8b73dbcc14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:08.630355 31438 consensus_queue.cc:1035] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "343aed55273742a3b112be9f5014e5da" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 34029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:08.630508 31438 consensus_queue.cc:1035] T a80dbe1943254cfca1b514baa181b786 P 8a32ee3dcbb14727b8af6a8b73dbcc14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bd1a231e7d4514ab5d8d261fe857dc" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:08.633831 31444 mvcc.cc:205] Tried to move back new op lower bound from 7299001747985956864 to 7299001747930234880. Current Snapshot: MvccSnapshot[applied={T|T < 7299001747985956864}]
I20260620 19:14:08.635941 31446 mvcc.cc:205] Tried to move back new op lower bound from 7299001747985956864 to 7299001747930234880. Current Snapshot: MvccSnapshot[applied={T|T < 7299001747985956864}]
I20260620 19:14:08.636427 31445 mvcc.cc:205] Tried to move back new op lower bound from 7299001747985956864 to 7299001747930234880. Current Snapshot: MvccSnapshot[applied={T|T < 7299001747985956864}]
I20260620 19:14:08.638677 30927 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:08.638767 30927 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:08.639503 30927 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b22195366264b649cad0a5129e9ff09: Sending DeleteTablet for 3 replicas of tablet a80dbe1943254cfca1b514baa181b786
I20260620 19:14:08.639850 31345 tablet_service.cc:1514] Processing DeleteTablet for tablet a80dbe1943254cfca1b514baa181b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:08 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:33262
I20260620 19:14:08.640228 31211 tablet_service.cc:1514] Processing DeleteTablet for tablet a80dbe1943254cfca1b514baa181b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:08 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:34416
I20260620 19:14:08.640475 31054 tablet_service.cc:1514] Processing DeleteTablet for tablet a80dbe1943254cfca1b514baa181b786 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:08 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55648
I20260620 19:14:08.641034 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 31009
I20260620 19:14:08.643039 31454 tablet_replica.cc:330] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da: stopping tablet replica
I20260620 19:14:08.643186 31454 raft_consensus.cc:2229] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:08.643344 31454 raft_consensus.cc:2258] T a80dbe1943254cfca1b514baa181b786 P 343aed55273742a3b112be9f5014e5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:08.643509 31453 tablet_replica.cc:330] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: stopping tablet replica
I20260620 19:14:08.643621 31453 raft_consensus.cc:2229] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:08.643729 31453 raft_consensus.cc:2258] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:08.644192 31453 ts_tablet_manager.cc:1905] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:08.645880 31453 ts_tablet_manager.cc:1918] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:08.646006 31453 log.cc:1196] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/a80dbe1943254cfca1b514baa181b786
I20260620 19:14:08.646989 31453 ts_tablet_manager.cc:1939] T a80dbe1943254cfca1b514baa181b786 P 09bd1a231e7d4514ab5d8d261fe857dc: Deleting consensus metadata
W20260620 19:14:08.647372 30915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:35343 (error 108)
I20260620 19:14:08.647517 30914 catalog_manager.cc:4841] TS 09bd1a231e7d4514ab5d8d261fe857dc (127.19.66.195:33021): tablet a80dbe1943254cfca1b514baa181b786 (table test-table [id=7b623a6033294e5eab2ee16589b25c83]) successfully deleted
I20260620 19:14:08.647931 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 31146
I20260620 19:14:08.653349 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.653188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:35343 (local address 127.0.0.1:55660)
0620 19:14:08.653269 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.653307 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260620 19:14:08.653465 30915 catalog_manager.cc:4584] TS 8a32ee3dcbb14727b8af6a8b73dbcc14 (127.19.66.193:35343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a80dbe1943254cfca1b514baa181b786: Network error: Client connection negotiation failed: client connection to 127.19.66.193:35343: connect: Connection refused (error 111)
I20260620 19:14:08.654268 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 31283
I20260620 19:14:08.654315 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:08.654229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:34029 (local address 127.0.0.1:34428)
0620 19:14:08.654271 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:08.654291 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.194:34029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:14:08.659304 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 30896
I20260620 19:14:08.664286 19723 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:14:08.998651 19723 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:14:08.998757 19723 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:14:09.333003 19723 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:14:09.333 UTC [30482] LOG: received smart shutdown request
2026-06-20 19:14:09.335 UTC [30482] LOG: background worker "logical replication launcher" (PID 30491) exited with exit code 1
2026-06-20 19:14:09.335 UTC [30486] LOG: shutting down
2026-06-20 19:14:09.468 UTC [30482] LOG: database system is shut down
2026-06-20T19:14:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42381 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:09 dist-test-slave-nq8r krb5kdc[31465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:09 dist-test-slave-nq8r krb5kdc[31465](info): set up 2 sockets
Jun 20 19:14:09 dist-test-slave-nq8r krb5kdc[31465](info): commencing operation
krb5kdc: starting...
W20260620 19:14:10.492568 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:14:10 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangeradmin_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangeradmin_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangerlookup_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangerlookup_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
I20260620 19:14:10.637228 19723 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskiAqWb6/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/postgres -l logfile start
2026-06-20 19:14:12.776 UTC [31495] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 19:14:12.776 UTC [31495] LOG: listening on IPv4 address "127.19.66.212", port 45969
2026-06-20 19:14:12.779 UTC [31495] LOG: listening on Unix socket "/tmp/.s.PGSQL.45969"
2026-06-20 19:14:12.806 UTC [31498] LOG: database system was shut down at 2026-06-20 19:14:11 UTC
2026-06-20 19:14:12.809 UTC [31495] LOG: database system is ready to accept connections
I20260620 19:14:13.688470 19723 mini_postgres.cc:96] Postgres bound to 45969
2026-06-20 19:14:13.694 UTC [31506] FATAL: database "slave" does not exist
127.19.66.212:45969 - accepting connections
I20260620 19:14:13.694497 19723 mini_ranger.cc:161] Starting Ranger...
I20260620 19:14:13.707878 19723 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 19:14:14.069629 19723 mini_ranger.cc:87] Created ranger Postgres database
I20260620 19:14:14.069694 19723 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin
2026-06-20 19:14:14,326 [I] DB FLAVOR :POSTGRES
2026-06-20 19:14:14,326 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:14:14,326 [I] Checking connection..
2026-06-20 19:14:14,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 19:14:14,554 [I] Checking connection passed.
2026-06-20 19:14:14,555 [I] --------- Verifying version history table ---------
2026-06-20 19:14:14,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:14:14,777 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:14:14,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:14:15,014 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 19:14:15,014 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 19:14:15,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 19:14:15.243 UTC [31570] WARNING: there is no transaction in progress
2026-06-20 19:14:15,248 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 19:14:15,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 19:14:15,488 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 19:14:15,489 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:14:15,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 19:14:15,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 19:14:15,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nq8r.c.gcp-upstream.internal','N') ;"
2026-06-20 19:14:16,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:14:16,465 [I] Table x_portal_user does not exist in database ranger
2026-06-20 19:14:16,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:14:16,699 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 19:14:16,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:14:16,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:14:17,185 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 19:14:17,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 19:14:18.039 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.058 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.072 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.086 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.100 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.115 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.164 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.169 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.181 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.190 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.198 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.210 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.218 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.224 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.229 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18.234 UTC [31678] WARNING: there is no transaction in progress
2026-06-20 19:14:18,763 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 19:14:18,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 19:14:18,984 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 19:14:18,984 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 19:14:19,216 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 19:14:19,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 19:14:19,446 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 19:14:19,446 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 19:14:19,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 19:14:19,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nq8r.c.gcp-upstream.internal';"
2026-06-20 19:14:20,171 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 19:14:20,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 19:14:20,420 [I] DB_PATCHES have already been applied
I20260620 19:14:20.425520 19723 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskiAqWb6/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 19:14:20.425581 19723 mini_ranger.cc:193] Using host: 127.19.66.212
I20260620 19:14:20.428408 19723 mini_ranger.cc:227] Ranger admin URL: http://127.19.66.212:43275
Jun 20, 2026 7:14:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:14:20 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:14:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 7:14:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 7:14:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 7:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 7:14:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 19:14:21.423071 19723 mini_ranger.cc:160] Time spent starting Ranger: real 7.729s user 0.000s sys 0.007s
Jun 20, 2026 7:14:39 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 7:14:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 7:14:39 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 7:14:39 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 7:14:40 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 19:14:41.249850 19723 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.19.66.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.19.66.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangerkms_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.19.66.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/rangerkms_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.19.66.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.19.66.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.19.66.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/HTTP_127.19.66.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 19:14:41 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 19:14:41.340493 19723 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 19:14:41.349710 19723 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 19:14:42.018584 19723 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 19:14:42.018663 19723 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms
I20260620 19:14:42.018678 19723 mini_ranger_kms.cc:227] Using postgres at 127.19.66.212:45969
2026-06-20 19:14:42,161 [I] DB FLAVOR :POSTGRES
2026-06-20 19:14:42,161 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 19:14:42,161 [I] Checking connection
2026-06-20 19:14:42,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 19:14:42,421 [I] connection success
2026-06-20 19:14:42,422 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 19:14:42,422 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 19:14:42,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 19:14:42,657 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 19:14:42,657 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 19:14:42,657 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 19:14:42,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.19.66.212:45969/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 19:14:42,898 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 19:14:43.338271 19723 mini_ranger_kms.cc:310] Created kms service
I20260620 19:14:43.524767 19723 mini_ranger_kms.cc:326] Created kudu user
I20260620 19:14:43.566946 19723 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 19:14:43 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982881, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:43.928756 19723 mini_ranger_kms.cc:384] Added ranger policy
I20260620 19:14:43.929031 19723 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms:/tmp/dist-test-taskiAqWb6/build/release/bin/postgresql.jar:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskiAqWb6/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 19:14:43.929059 19723 mini_ranger_kms.cc:240] Using host: 127.19.66.212
I20260620 19:14:43.932616 19723 mini_ranger_kms.cc:276] Ranger KMS PID: 31858
I20260620 19:14:43.932677 19723 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.19.66.212:55113
Jun 20, 2026 7:14:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 7:14:44 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 7:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 7:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 7:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 7:14:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 19:14:45.932493 19723 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.592s user 0.002s sys 0.124s
I20260620 19:14:45.932592 19723 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 19:14:45.932637 19723 mini_ranger_kms.cc:401] 127.19.66.212:55113/kms/v1/keys
Jun 20 19:14:45 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 19:14:46 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
Jun 20 19:14:46 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982885, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.19.66.212@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
Jun 20 19:14:48 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:14:49.023260 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38311
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38311
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:49.129254 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:49.129469 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:49.129491 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:49.130733 31909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:49.130779 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:49.130792 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:49.130805 31909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:49.130815 31909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:49.132340 31909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
--ranger_kms_url=127.19.66.212:55113/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38311
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:49.132572 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:49.132782 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:49.138926 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:49.139035 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:49.138901 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:49.139060 31909 server_base.cc:1034] running on GCE node
I20260620 19:14:49.139525 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:49.139840 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:49.141014 31909 hybrid_clock.cc:648] HybridClock initialized: now 1781982889140999 us; error 33 us; skew 500 ppm
Jun 20 19:14:49 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982889, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:49.146497 31909 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:49.147032 31909 webserver.cc:460] Webserver started at http://127.19.66.254:37891/ using document root <none> and password file <none>
I20260620 19:14:49.147244 31909 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:49.147284 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:49.147413 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:49 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982889, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:49.194636 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "0a07624a33ac4036a176e90f4a35b728"
format_stamp: "Formatted at 2026-06-20 19:14:49 on dist-test-slave-nq8r"
server_key: "d81ad301b2b8b3fd67c94a1232a067bb"
server_key_iv: "8a2654d10a5ab21205a3ebcc2ea413dc"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:49.195189 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "0a07624a33ac4036a176e90f4a35b728"
format_stamp: "Formatted at 2026-06-20 19:14:49 on dist-test-slave-nq8r"
server_key: "d81ad301b2b8b3fd67c94a1232a067bb"
server_key_iv: "8a2654d10a5ab21205a3ebcc2ea413dc"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:49.197057 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:49.212962 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:49.213181 31909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:49.213291 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "0a07624a33ac4036a176e90f4a35b728"
format_stamp: "Formatted at 2026-06-20 19:14:49 on dist-test-slave-nq8r"
server_key: "d81ad301b2b8b3fd67c94a1232a067bb"
server_key_iv: "8a2654d10a5ab21205a3ebcc2ea413dc"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:49.213400 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:49.221688 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:49.223392 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:49.223523 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:49.227901 31909 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:38311
I20260620 19:14:49.227958 31976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:38311 every 8 connection(s)
I20260620 19:14:49.228297 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:49.228971 31977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:49.229224 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 31909
I20260620 19:14:49.229308 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:14:49.229487 19723 external_mini_cluster.cc:1428] Setting key f230f92b989299d74de36038188a4d91
I20260620 19:14:49.231431 31977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Bootstrap starting.
I20260620 19:14:49.232049 31977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:49.232353 31977 log.cc:824] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:49.233109 31977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: No bootstrap required, opened a new log
I20260620 19:14:49.235282 31977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:49.235417 31977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:49.235450 31977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a07624a33ac4036a176e90f4a35b728, State: Initialized, Role: FOLLOWER
I20260620 19:14:49.235615 31977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:49.235692 31977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:49.235741 31977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:49.235790 31977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:49.236297 31977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:49.236393 31977 leader_election.cc:302] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a07624a33ac4036a176e90f4a35b728; no voters:
I20260620 19:14:49.236572 31977 leader_election.cc:288] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:49.236613 31982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:49.236801 31982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 LEADER]: Becoming Leader. State: Replica: 0a07624a33ac4036a176e90f4a35b728, State: Running, Role: LEADER
I20260620 19:14:49.236932 31977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:49.236953 31982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:49.237326 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a07624a33ac4036a176e90f4a35b728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } } }
I20260620 19:14:49.237466 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:49.237650 31982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a07624a33ac4036a176e90f4a35b728. Latest consensus state: current_term: 1 leader_uuid: "0a07624a33ac4036a176e90f4a35b728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } } }
I20260620 19:14:49.237749 31982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:49.237924 31991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:49.238432 31991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:49.238766 31977 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Jun 20 19:14:49 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:49.246274 31980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:49.229976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.19.66.254:38311)
0620 19:14:49.230176 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:49.230187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:49.231757 (+ 1570us) server_negotiation.cc:407] Connection header received
0620 19:14:49.232330 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:49.232339 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:49.232402 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:49.232571 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:49.233315 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:49.234682 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:49.235757 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:49.236382 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:49.236458 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:49.240745 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:49.240790 (+ 45us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:49.240791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:49.240923 (+ 132us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:49.243727 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:49.244182 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:49.244185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:49.244185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:49.244298 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:49.244520 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:49.244521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:49.244521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:49.244901 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:49.245085 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:49.245178 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 19:14:49.245931 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":79,"threads_started":1}
I20260620 19:14:49.246872 31991 catalog_manager.cc:1347] Generated new cluster ID: 1ad8120b34a946cda26cea9e7a85ca20
I20260620 19:14:49.246955 31991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:49.258457 31991 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:49.259243 31991 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:49.266551 31991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Generated new TSK 0
I20260620 19:14:49.266848 31991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:14:49.928364 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:50.011236 32022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:50.011449 32022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:50.011479 32022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:50.012701 32022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:50.012741 32022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:50.012777 32022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:14:50.014297 32022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:50.014544 32022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:50.014812 32022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:50.019649 32027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.019676 32028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.019687 32030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.020126 32022 server_base.cc:1034] running on GCE node
I20260620 19:14:50.020318 32022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:50.020542 32022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:50.021678 32022 hybrid_clock.cc:648] HybridClock initialized: now 1781982890021659 us; error 38 us; skew 500 ppm
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.024151 32022 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:14:50.024578 32022 webserver.cc:460] Webserver started at http://127.19.66.193:45953/ using document root <none> and password file <none>
I20260620 19:14:50.024807 32022 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:50.024883 32022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:50.025008 32022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:50.038589 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "d788449969e1498abba96140b0fdedb7"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "376fd1830ce6eb30b45b8b461a8403bb"
server_key_iv: "cb0e778397d077d39c91ba4665033e1a"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.045399 32022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "d788449969e1498abba96140b0fdedb7"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "376fd1830ce6eb30b45b8b461a8403bb"
server_key_iv: "cb0e778397d077d39c91ba4665033e1a"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.046993 32022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:14:50.059952 32037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.060118 32022 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:50.060210 32022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "d788449969e1498abba96140b0fdedb7"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "376fd1830ce6eb30b45b8b461a8403bb"
server_key_iv: "cb0e778397d077d39c91ba4665033e1a"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.060298 32022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:50.085053 32022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:50.086951 32022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:50.087090 32022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:50.087301 32022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:50.087597 32022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:50.087630 32022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.087661 32022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:50.087683 32022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.094141 32022 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:45915
I20260620 19:14:50.094254 32152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:45915 every 8 connection(s)
I20260620 19:14:50.094633 32022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:50.104286 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32022
I20260620 19:14:50.104379 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:14:50.104564 19723 external_mini_cluster.cc:1428] Setting key 1d45fba926ccc11a9e71a16c30ae2991
I20260620 19:14:50.105754 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.095455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:40763 (local address 127.19.66.254:38311)
0620 19:14:50.095603 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:50.095605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:50.095830 (+ 225us) server_negotiation.cc:407] Connection header received
0620 19:14:50.096534 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:50.096535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:50.096550 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:50.096620 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:50.097371 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.098224 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.099690 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.099887 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.099942 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.102217 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:50.102219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:50.102219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:50.102227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:50.104047 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.104556 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.104557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.104558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.104628 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.104939 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.104941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.104941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.105114 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:50.105159 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:50.105567 (+ 408us) server_negotiation.cc:299] Negotiation successful
0620 19:14:50.105659 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260620 19:14:50.105887 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.095430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.193:40763)
0620 19:14:50.095673 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:50.095720 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:50.096406 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:50.096703 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:50.096708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:50.096979 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:50.097270 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.097276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.098324 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.098325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.099601 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.099604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.099982 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.099983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.100135 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.100655 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:50.100665 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:50.102122 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:50.104150 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.104157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.104160 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.104458 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.104699 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.104699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.104700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.104836 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.105209 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:50.105214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:50.105394 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 19:14:50.105502 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 19:14:50.105663 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260620 19:14:50.106411 32153 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:50.106503 32153 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.106699 32153 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:50.107214 31941 ts_manager.cc:194] Registered new tserver with Master: d788449969e1498abba96140b0fdedb7 (127.19.66.193:45915)
I20260620 19:14:50.107968 31941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:40763
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:14:50.127681 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:50.231221 32159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:50.231417 32159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:50.231448 32159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:50.232596 32159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:50.232640 32159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:50.232673 32159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:14:50.234120 32159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:50.234334 32159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:50.234544 32159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:50.239470 32169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.239485 32167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.239838 32166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.239871 32159 server_base.cc:1034] running on GCE node
I20260620 19:14:50.240365 32159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:50.240597 32159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:50.241732 32159 hybrid_clock.cc:648] HybridClock initialized: now 1781982890241719 us; error 29 us; skew 500 ppm
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.244076 32159 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:14:50.244503 32159 webserver.cc:460] Webserver started at http://127.19.66.194:38743/ using document root <none> and password file <none>
I20260620 19:14:50.244715 32159 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:50.244761 32159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:50.244922 32159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:50.256989 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "8c0b2e3e21dd44fd806065383647036b"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "175da4545d8f61820c281760f554a86a"
server_key_iv: "54ab2e4813eb78364c6e0f9998f66d6f"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.257498 32159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "8c0b2e3e21dd44fd806065383647036b"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "175da4545d8f61820c281760f554a86a"
server_key_iv: "54ab2e4813eb78364c6e0f9998f66d6f"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.259289 32159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.276963 32176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.288992 32159 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.001s sys 0.000s
I20260620 19:14:50.289148 32159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "8c0b2e3e21dd44fd806065383647036b"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "175da4545d8f61820c281760f554a86a"
server_key_iv: "54ab2e4813eb78364c6e0f9998f66d6f"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.289261 32159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:50.313151 32159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:50.315706 32159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:50.315896 32159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:50.316182 32159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:50.316581 32159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:50.316670 32159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.316748 32159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:50.316802 32159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.323995 32159 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:33907
I20260620 19:14:50.324537 32159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:50.330446 32289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:33907 every 8 connection(s)
I20260620 19:14:50.334784 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32159
I20260620 19:14:50.334872 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:14:50.335079 19723 external_mini_cluster.cc:1428] Setting key 3d778e7e77a54ba826023d4adf7e8240
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:50.343823 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.330980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:59877 (local address 127.19.66.254:38311)
0620 19:14:50.331047 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:50.331050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:50.331353 (+ 303us) server_negotiation.cc:407] Connection header received
0620 19:14:50.332017 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:50.332019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:50.332032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:50.332059 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:50.332892 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.333657 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.337410 (+ 3753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.337751 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.337802 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.340579 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:50.340580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:50.340580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:50.340588 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:50.342192 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.342696 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.342698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.342698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.342758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.343042 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.343043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.343044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.343192 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:50.343251 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:50.343631 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 19:14:50.343724 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:14:50.344556 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.330961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.194:59877)
0620 19:14:50.331214 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:50.331255 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:50.331894 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:50.332241 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:50.332245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:50.332479 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:50.332765 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.332772 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.333790 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.333792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.335121 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.335124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.337947 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.337948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.338097 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.338815 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:50.338826 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:50.340449 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:50.342287 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.342294 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.342297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.342603 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.342825 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.342826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.342826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.342959 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.343288 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:50.343294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:50.343462 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 19:14:50.344158 (+ 696us) client_negotiation.cc:240] Negotiation successful
0620 19:14:50.344321 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":78,"threads_started":1}
I20260620 19:14:50.345229 32290 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:50.345369 32290 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.345611 32290 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:50.346001 31941 ts_manager.cc:194] Registered new tserver with Master: 8c0b2e3e21dd44fd806065383647036b (127.19.66.194:33907)
I20260620 19:14:50.346480 31941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:59877
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:14:50.364660 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:50.484874 32296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:50.485221 32296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:50.485301 32296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:50.487090 32296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:50.487191 32296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:50.487267 32296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:14:50.489424 32296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:50.489876 32296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:50.490177 32296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:50.496770 32304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.497936 32296 server_base.cc:1034] running on GCE node
W20260620 19:14:50.498152 32302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.501009 32301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.501353 32296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:50.501645 32296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:50.502800 32296 hybrid_clock.cc:648] HybridClock initialized: now 1781982890502780 us; error 35 us; skew 500 ppm
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.506050 32296 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:14:50.506549 32296 webserver.cc:460] Webserver started at http://127.19.66.195:36717/ using document root <none> and password file <none>
I20260620 19:14:50.506821 32296 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:50.506868 32296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:50.507047 32296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:50.525614 32296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "9efeffef35924f1b821abf1cfa783f75"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "ee553068b33c2547684c1e9837686997"
server_key_iv: "a82dbe2c722b346012a9361a811bc9a2"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.526137 32296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "9efeffef35924f1b821abf1cfa783f75"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "ee553068b33c2547684c1e9837686997"
server_key_iv: "a82dbe2c722b346012a9361a811bc9a2"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.529122 32296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260620 19:14:50.546221 32311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.546458 32296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:50.546582 32296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "9efeffef35924f1b821abf1cfa783f75"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "ee553068b33c2547684c1e9837686997"
server_key_iv: "a82dbe2c722b346012a9361a811bc9a2"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.546727 32296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:50.576645 32296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:50.579062 32296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:50.579203 32296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:50.579420 32296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:50.579823 32296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:50.579892 32296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.579945 32296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:50.579986 32296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.586767 32296 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:43349
I20260620 19:14:50.586881 32424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:43349 every 8 connection(s)
I20260620 19:14:50.587216 32296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:50.594983 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32296
I20260620 19:14:50.595082 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:14:50.595248 19723 external_mini_cluster.cc:1428] Setting key c47f1a4299160f6d426634b21d4243bd
I20260620 19:14:50.597849 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.588226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:35367 (local address 127.19.66.254:38311)
0620 19:14:50.588273 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:50.588275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:50.588285 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:14:50.588873 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:50.588875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:50.588887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:50.588950 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:50.589622 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.590331 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.591726 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.592031 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.592073 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.594627 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:50.594628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:50.594629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:50.594637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:50.596338 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.596781 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.596782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.596783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.596839 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.597122 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.597122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.597123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.597251 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:50.597292 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:50.597667 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 19:14:50.597749 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:14:50.598649 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.587942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.195:35367)
0620 19:14:50.588106 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:50.588153 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:50.588734 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:50.589038 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:50.589042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:50.589263 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:50.589518 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.589524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.590423 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.590425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.591627 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.591630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.592175 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.592176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.592310 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.592966 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:50.592976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:50.594527 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:50.596421 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.596427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.596429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.596705 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.596925 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.596925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.596926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.597052 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.597335 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:50.597342 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:50.597506 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 19:14:50.598194 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 19:14:50.598379 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260620 19:14:50.599220 32425 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:50.599323 32425 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.599510 32425 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:50.599931 31941 ts_manager.cc:194] Registered new tserver with Master: 9efeffef35924f1b821abf1cfa783f75 (127.19.66.195:43349)
I20260620 19:14:50.600412 31941 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:35367
I20260620 19:14:50.609496 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:50.610167 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32022
I20260620 19:14:50.616585 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32159
I20260620 19:14:50.621435 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32296
I20260620 19:14:50.630587 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 31909
I20260620 19:14:50.637277 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38311
--webserver_interface=127.19.66.254
--webserver_port=37891
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38311
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:50.715731 32428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:50.715916 32428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:50.715936 32428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:50.717007 32428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:50.717039 32428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:50.717052 32428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:50.717062 32428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:50.717073 32428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:50.718565 32428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38311
--ranger_config_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:50.718736 32428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:50.718885 32428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:50.723151 32436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.723263 32428 server_base.cc:1034] running on GCE node
W20260620 19:14:50.723135 32433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.723189 32434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.723541 32428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:50.723805 32428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:50.724887 32428 hybrid_clock.cc:648] HybridClock initialized: now 1781982890724873 us; error 27 us; skew 500 ppm
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.727065 32428 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:50.727404 32428 webserver.cc:460] Webserver started at http://127.19.66.254:37891/ using document root <none> and password file <none>
I20260620 19:14:50.727631 32428 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:50.727733 32428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:50.766970 32428 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:50.767680 32444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.767856 32428 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:50.767962 32428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "0a07624a33ac4036a176e90f4a35b728"
format_stamp: "Formatted at 2026-06-20 19:14:49 on dist-test-slave-nq8r"
server_key: "d81ad301b2b8b3fd67c94a1232a067bb"
server_key_iv: "8a2654d10a5ab21205a3ebcc2ea413dc"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.768428 32428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:50.777456 32428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:50.778920 32428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:50.779021 32428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:50.782776 32428 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:38311
I20260620 19:14:50.782845 32496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:38311 every 8 connection(s)
I20260620 19:14:50.783159 32428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:50.784335 32497 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:14:50.784969 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Bootstrap starting.
I20260620 19:14:50.792456 32497 log.cc:824] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:50.792883 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32428
I20260620 19:14:50.793157 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:45915
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:38311
--webserver_port=45953
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:50.794571 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:14:50.794806 32497 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Bootstrap complete.
I20260620 19:14:50.797775 32497 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:50.798138 32497 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a07624a33ac4036a176e90f4a35b728, State: Initialized, Role: FOLLOWER
I20260620 19:14:50.798317 32497 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:50.798389 32497 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:50.798439 32497 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:50.798486 32497 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:14:50.799835 32497 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:50.799949 32497 leader_election.cc:302] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a07624a33ac4036a176e90f4a35b728; no voters:
I20260620 19:14:50.800136 32497 leader_election.cc:288] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:14:50.800185 32501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:14:50.800348 32501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [term 2 LEADER]: Becoming Leader. State: Replica: 0a07624a33ac4036a176e90f4a35b728, State: Running, Role: LEADER
I20260620 19:14:50.800391 32497 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:50.800499 32501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } }
I20260620 19:14:50.800812 32502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0a07624a33ac4036a176e90f4a35b728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } } }
I20260620 19:14:50.800815 32503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a07624a33ac4036a176e90f4a35b728. Latest consensus state: current_term: 2 leader_uuid: "0a07624a33ac4036a176e90f4a35b728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a07624a33ac4036a176e90f4a35b728" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38311 } } }
I20260620 19:14:50.800928 32502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:50.800933 32503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:50.801100 32508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:50.801561 32508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:50.801719 32508 catalog_manager.cc:1259] Loaded cluster ID: 1ad8120b34a946cda26cea9e7a85ca20
I20260620 19:14:50.801754 32508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:50.801932 32497 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 19:14:50.805003 32508 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:50.805516 32508 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Loaded TSK: 0
I20260620 19:14:50.805785 32508 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:14:50.877424 32500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:50.877669 32500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:50.877703 32500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:50.879359 32500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:50.879400 32500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:50.879434 32500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:14:50.881523 32500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:45915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=45953
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:50.881749 32500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:50.881995 32500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:50.888114 32534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.888336 32537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:50.888620 32535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:50.889042 32500 server_base.cc:1034] running on GCE node
I20260620 19:14:50.889238 32500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:50.889487 32500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:50.890626 32500 hybrid_clock.cc:648] HybridClock initialized: now 1781982890890606 us; error 32 us; skew 500 ppm
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:50.893779 32500 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:14:50.894357 32500 webserver.cc:460] Webserver started at http://127.19.66.193:45953/ using document root <none> and password file <none>
I20260620 19:14:50.894651 32500 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:50.894781 32500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:50.917299 32500 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:50.920729 32545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.920997 32500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260620 19:14:50.921089 32500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "d788449969e1498abba96140b0fdedb7"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "376fd1830ce6eb30b45b8b461a8403bb"
server_key_iv: "cb0e778397d077d39c91ba4665033e1a"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:50.921553 32500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:50.949072 32500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:50.951253 32500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:50.951403 32500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:50.951656 32500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:50.952046 32500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:50.952104 32500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.952136 32500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:50.952162 32500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:50.958930 32500 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:45915
I20260620 19:14:50.959372 32500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:14:50.959838 32658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:45915 every 8 connection(s)
Jun 20 19:14:50 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982890, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:50.969535 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32500
I20260620 19:14:50.969830 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:33907
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:38311
--webserver_port=38743
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:50.979455 32661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.960150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.193:41031)
0620 19:14:50.960336 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:50.960389 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:50.960934 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:50.961528 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:50.961532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:50.961754 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:50.962026 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.962032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.963235 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.963237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.964121 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:50.964124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.965099 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.965100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:50.965241 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.965813 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:50.965823 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:50.967976 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:50.972029 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.972038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.972041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.972348 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.972588 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:50.972589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:50.972589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:50.972727 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:50.978054 (+ 5327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:50.978066 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:50.978255 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 19:14:50.978434 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 19:14:50.979125 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
I20260620 19:14:50.980659 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:50.960232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:41031 (local address 127.19.66.254:38311)
0620 19:14:50.960449 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:50.960456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:50.960492 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:14:50.961169 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:50.961175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:50.961247 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:50.961376 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:50.962141 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.963115 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.964200 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:50.964834 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:50.964972 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:50.968085 (+ 3113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:50.968090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:50.968090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:50.968102 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:50.971923 (+ 3821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.972431 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.972433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.972433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.972525 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:50.972794 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:50.972795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:50.972796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:50.977903 (+ 5107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:50.979626 (+ 1723us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:50.979714 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:14:50.980438 (+ 724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":99,"threads_started":1}
I20260620 19:14:50.981370 32659 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:50.981487 32659 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.981701 32659 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:50.982723 32659 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:38311 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:14:50.982800 32659 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.982867 32659 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:50.983215 32659 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:50.983294 32659 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:50.983601 32659 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:14:51.080039 32663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:51.080344 32663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:51.080402 32663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:51.082194 32663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:51.082305 32663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:51.082391 32663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:14:51.084679 32663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:33907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=38743
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:51.085119 32663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:51.085438 32663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:51.091115 32668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:51.091248 32669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:51.092631 32663 server_base.cc:1034] running on GCE node
W20260620 19:14:51.092911 32671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:51.093210 32663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:51.093546 32663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:51.094718 32663 hybrid_clock.cc:648] HybridClock initialized: now 1781982891094696 us; error 35 us; skew 500 ppm
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:51.097483 32663 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:14:51.097962 32663 webserver.cc:460] Webserver started at http://127.19.66.194:38743/ using document root <none> and password file <none>
I20260620 19:14:51.098198 32663 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:51.098258 32663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:51.121055 32663 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260620 19:14:51.122152 32679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.122363 32663 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:51.122450 32663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "8c0b2e3e21dd44fd806065383647036b"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "175da4545d8f61820c281760f554a86a"
server_key_iv: "54ab2e4813eb78364c6e0f9998f66d6f"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:51.122913 32663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:51.136080 32663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:51.138732 32663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:51.138927 32663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:51.139194 32663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:51.139614 32663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:51.139714 32663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.139784 32663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:51.139838 32663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.146741 32663 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:33907
I20260620 19:14:51.147274 32663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:51.147943 324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:33907 every 8 connection(s)
I20260620 19:14:51.148298 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 32663
I20260620 19:14:51.148548 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.19.66.212:55113/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:43349
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:38311
--webserver_port=36717
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:51.175380 328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:51.150884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.194:37351)
0620 19:14:51.152999 (+ 2115us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:51.153050 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:51.165465 (+ 12415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:51.165840 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:51.165845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:51.166088 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:51.166396 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:51.166404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.167433 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.167434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:51.168717 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:51.168719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.169343 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.169345 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:51.169499 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:51.170129 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:51.170140 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:51.171769 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:51.173701 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:51.173708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:51.173711 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:51.174009 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:51.174227 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:51.174228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:51.174229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:51.174366 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:51.174685 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:51.174692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:51.174861 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 19:14:51.174976 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 19:14:51.175140 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2065,"thread_start_us":74,"threads_started":1}
I20260620 19:14:51.175963 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:51.151172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37351 (local address 127.19.66.254:38311)
0620 19:14:51.151234 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:51.151236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:51.153156 (+ 1920us) server_negotiation.cc:407] Connection header received
0620 19:14:51.165622 (+ 12466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:51.165625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:51.165644 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:51.165756 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:51.166521 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.167322 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.168818 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.169171 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.169247 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:51.171877 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:51.171879 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:51.171880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:51.171888 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:51.173610 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.174095 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.174096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.174096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.174158 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.174442 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.174443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.174444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.174596 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:51.174658 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:51.175726 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0620 19:14:51.175811 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:14:51.176816 325 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:51.176934 325 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.177152 325 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:51.177882 325 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:38311 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:14:51.177973 325 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.178035 325 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:51.178382 325 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.178453 325 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:51.178797 325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:14:51.271862 327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:51.272215 327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:51.272300 327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:51.274062 327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:51.274180 327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:51.274263 327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:14:51.276399 327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.19.66.212:55113/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:43349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=36717
--tserver_master_addrs=127.19.66.254:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:51.276862 327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:51.277182 327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:51.287050 336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:51.287472 333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:51.287658 327 server_base.cc:1034] running on GCE node
W20260620 19:14:51.287817 334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:51.288070 327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:51.288350 327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:51.289501 327 hybrid_clock.cc:648] HybridClock initialized: now 1781982891289484 us; error 33 us; skew 500 ppm
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:51.299224 327 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:14:51.299846 327 webserver.cc:460] Webserver started at http://127.19.66.195:36717/ using document root <none> and password file <none>
I20260620 19:14:51.300118 327 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:51.300199 327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:51.321933 327 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260620 19:14:51.322975 345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.323212 327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:51.323316 327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "9efeffef35924f1b821abf1cfa783f75"
format_stamp: "Formatted at 2026-06-20 19:14:50 on dist-test-slave-nq8r"
server_key: "ee553068b33c2547684c1e9837686997"
server_key_iv: "a82dbe2c722b346012a9361a811bc9a2"
server_key_version: "kuduclusterkey@0"
I20260620 19:14:51.323803 327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:51.361191 327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:51.363427 327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:51.363601 327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:51.363874 327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:51.364249 327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:51.364302 327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.364337 327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:51.364359 327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:51.371318 327 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:43349
I20260620 19:14:51.371394 458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:43349 every 8 connection(s)
I20260620 19:14:51.371737 327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:51.380095 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 327
I20260620 19:14:51.381105 32662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:51.372450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:47959 (local address 127.19.66.254:38311)
0620 19:14:51.372509 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:51.372511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:51.372735 (+ 224us) server_negotiation.cc:407] Connection header received
0620 19:14:51.373220 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:51.373222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:51.373234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:51.373290 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:51.374004 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.374809 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.375771 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.376068 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.376143 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:51.378043 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:51.378044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:51.378044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:51.378051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:51.379434 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.379790 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.379791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.379791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.379853 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.380124 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.380125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.380126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.380256 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:51.380324 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:51.380934 (+ 610us) server_negotiation.cc:299] Negotiation successful
0620 19:14:51.381014 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:14:51.381826 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:51.372456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38311 (local address 127.19.66.195:47959)
0620 19:14:51.372619 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:51.372652 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:51.373127 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:51.373371 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:51.373375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:51.373623 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:51.373887 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:51.373894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.374892 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.374893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:51.375691 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:51.375694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.376154 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.376155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:51.376251 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:51.376712 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:51.376719 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:51.377959 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:51.379506 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:51.379511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:51.379513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:51.379722 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:51.379925 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:51.379925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:51.379926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:51.380055 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:51.380573 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:51.380580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:51.380762 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 19:14:51.381427 (+ 665us) client_negotiation.cc:240] Negotiation successful
0620 19:14:51.381582 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260620 19:14:51.382244 459 heartbeater.cc:347] Connected to a master server at 127.19.66.254:38311
I20260620 19:14:51.382361 459 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.382524 459 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:51.383131 459 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:38311 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 19:14:51.383204 459 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.383268 459 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:51.383644 459 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.383720 459 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
W20260620 19:14:51.384142 459 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:14:51.388724 463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:51.380474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56032 (local address 127.19.66.254:38311)
0620 19:14:51.380739 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:51.380741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:51.380747 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:14:51.380765 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:51.380765 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:51.380776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:51.381557 (+ 781us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:51.381588 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.382255 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.383427 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:51.383722 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:51.383789 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:51.385037 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:51.385038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:51.385038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:51.385044 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:51.386624 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.387987 (+ 1363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.387989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.387989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.388051 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:51.388212 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:51.388213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:51.388213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:51.388329 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:51.388483 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:51.388545 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:14:51.388622 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":73,"threads_started":1}
Jun 20 19:14:51 dist-test-slave-nq8r krb5kdc[31465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:51.983804 32659 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:51.983979 32659 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:51.984421 32461 ts_manager.cc:194] Registered new tserver with Master: d788449969e1498abba96140b0fdedb7 (127.19.66.193:45915)
I20260620 19:14:51.985502 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:41031
I20260620 19:14:52.178994 325 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:52.179159 325 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:52.179611 32461 ts_manager.cc:194] Registered new tserver with Master: 8c0b2e3e21dd44fd806065383647036b (127.19.66.194:33907)
I20260620 19:14:52.180136 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:37351
Jun 20 19:14:52 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:52.384333 459 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:52.384486 459 heartbeater.cc:511] Master 127.19.66.254:38311 requested a full tablet report, sending...
I20260620 19:14:52.384877 32461 ts_manager.cc:194] Registered new tserver with Master: 9efeffef35924f1b821abf1cfa783f75 (127.19.66.195:43349)
I20260620 19:14:52.385399 32461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:47959
I20260620 19:14:52.391685 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:52.401974 480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.392704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56042 (local address 127.19.66.254:38311)
0620 19:14:52.393081 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.393083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.393200 (+ 117us) server_negotiation.cc:407] Connection header received
0620 19:14:52.393298 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.393299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.393311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.393356 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:52.394087 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.394829 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.395855 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.396185 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.396231 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.397727 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:52.397729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:52.397729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:52.397736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:52.400518 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:52.400956 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:52.400958 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:52.400958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:52.401051 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:52.401281 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:52.401282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:52.401282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:52.401422 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:52.401509 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.401749 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.401837 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":66,"threads_started":1}
I20260620 19:14:52.404618 32461 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:52.406992 32461 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:52.435139 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:45915 (local address 127.0.0.1:55580)
0620 19:14:52.423391 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.423433 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:52.423498 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:52.423886 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:52.423889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:52.423891 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:52.423960 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.423974 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.430706 (+ 6732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.430707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.431778 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.431781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.434812 (+ 3031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.434813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.434898 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.434913 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:14:52.434995 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 19:14:52.435050 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"thread_start_us":74,"threads_started":1}
I20260620 19:14:52.436635 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33907 (local address 127.0.0.1:45020)
0620 19:14:52.424078 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.424087 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:52.424121 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:52.425435 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:52.425436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:52.425438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:52.425485 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.425487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.432335 (+ 6848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.432336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.433770 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.433773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.436397 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.436398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.436469 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.436473 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:52.436517 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:14:52.436555 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1624,"thread_start_us":75,"threads_started":1}
I20260620 19:14:52.438670 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:43349 (local address 127.0.0.1:40454)
0620 19:14:52.424188 (+ 1586us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.424195 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:52.424234 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:52.425554 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:52.425555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:52.425556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:52.425585 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.425586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.426783 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.426784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.436784 (+ 10000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.436788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.438442 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.438443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.438510 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.438513 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:14:52.438554 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:14:52.438593 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":69,"threads_started":1}
I20260620 19:14:52.439109 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40454 (local address 127.19.66.195:43349)
0620 19:14:52.423322 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.423327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.424288 (+ 961us) server_negotiation.cc:407] Connection header received
0620 19:14:52.424327 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.424329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.424344 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.424408 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:52.425656 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.426680 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.436884 (+ 10204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.438007 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.438074 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.438122 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.438986 (+ 864us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.439032 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":49,"threads_started":1}
I20260620 19:14:52.439571 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.19.66.193:45915)
0620 19:14:52.423648 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.423653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.423663 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:14:52.423697 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.423699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.423716 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.423790 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:52.424505 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.430609 (+ 6104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.433851 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.434641 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.434712 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.434776 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.439447 (+ 4671us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.439498 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1315,"thread_start_us":55,"threads_started":1}
I20260620 19:14:52.439908 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.422831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.19.66.194:33907)
0620 19:14:52.425215 (+ 2384us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.425220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.425229 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:14:52.425260 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.425262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.425277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.425350 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:52.427078 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.432240 (+ 5162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.435453 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.436244 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.436312 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.436364 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.439793 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.439839 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2348,"thread_start_us":51,"threads_started":1}
I20260620 19:14:52.439991 393 tablet_service.cc:1467] Processing CreateTablet for tablet 94499ff5f1814c8d868765d61734d02c (DEFAULT_TABLE table=test-table [id=783ade3703da43c69736ea9dcd0adf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:52.440377 393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94499ff5f1814c8d868765d61734d02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:52.440361 32575 tablet_service.cc:1467] Processing CreateTablet for tablet 94499ff5f1814c8d868765d61734d02c (DEFAULT_TABLE table=test-table [id=783ade3703da43c69736ea9dcd0adf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:52.440678 32575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94499ff5f1814c8d868765d61734d02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:52.440601 32709 tablet_service.cc:1467] Processing CreateTablet for tablet 94499ff5f1814c8d868765d61734d02c (DEFAULT_TABLE table=test-table [id=783ade3703da43c69736ea9dcd0adf28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:52.440910 32709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94499ff5f1814c8d868765d61734d02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:52.446869 494 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Bootstrap starting.
I20260620 19:14:52.447551 494 tablet_bootstrap.cc:654] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:52.447911 494 log.cc:824] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:52.448992 495 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Bootstrap starting.
I20260620 19:14:52.449136 493 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Bootstrap starting.
I20260620 19:14:52.449855 493 tablet_bootstrap.cc:654] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:52.450207 493 log.cc:824] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:52.450568 495 tablet_bootstrap.cc:654] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:52.451231 495 log.cc:824] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:52.457974 494 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: No bootstrap required, opened a new log
I20260620 19:14:52.458127 494 ts_tablet_manager.cc:1397] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260620 19:14:52.460377 494 raft_consensus.cc:348] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.460747 494 raft_consensus.cc:374] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:52.460868 494 raft_consensus.cc:729] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d788449969e1498abba96140b0fdedb7, State: Initialized, Role: FOLLOWER
I20260620 19:14:52.461078 494 consensus_queue.cc:260] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.461453 494 ts_tablet_manager.cc:1428] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:52.461570 495 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: No bootstrap required, opened a new log
I20260620 19:14:52.461628 32659 heartbeater.cc:503] Master 127.19.66.254:38311 was elected leader, sending a full tablet report...
I20260620 19:14:52.461655 495 ts_tablet_manager.cc:1397] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
W20260620 19:14:52.461874 32660 tablet.cc:2369] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:52.462433 493 tablet_bootstrap.cc:492] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: No bootstrap required, opened a new log
I20260620 19:14:52.462522 493 ts_tablet_manager.cc:1397] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.000s
I20260620 19:14:52.465742 495 raft_consensus.cc:348] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.466388 495 raft_consensus.cc:374] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:52.466483 495 raft_consensus.cc:729] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c0b2e3e21dd44fd806065383647036b, State: Initialized, Role: FOLLOWER
I20260620 19:14:52.467154 495 consensus_queue.cc:260] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.467504 495 ts_tablet_manager.cc:1428] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260620 19:14:52.467705 493 raft_consensus.cc:348] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.467898 493 raft_consensus.cc:374] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:52.467969 493 raft_consensus.cc:729] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efeffef35924f1b821abf1cfa783f75, State: Initialized, Role: FOLLOWER
I20260620 19:14:52.468027 325 heartbeater.cc:503] Master 127.19.66.254:38311 was elected leader, sending a full tablet report...
I20260620 19:14:52.468124 493 consensus_queue.cc:260] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.468369 493 ts_tablet_manager.cc:1428] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260620 19:14:52.468484 459 heartbeater.cc:503] Master 127.19.66.254:38311 was elected leader, sending a full tablet report...
Jun 20 19:14:52 dist-test-slave-nq8r krb5kdc[31465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982891, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for HTTP/127.19.66.212@KRBTEST.COM
I20260620 19:14:52.610612 499 raft_consensus.cc:482] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:52.610783 499 raft_consensus.cc:504] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.611203 499 leader_election.cc:288] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c0b2e3e21dd44fd806065383647036b (127.19.66.194:33907), 9efeffef35924f1b821abf1cfa783f75 (127.19.66.195:43349)
I20260620 19:14:52.615389 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.611520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:46761 (local address 127.19.66.194:33907)
0620 19:14:52.611586 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.611588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.611717 (+ 129us) server_negotiation.cc:407] Connection header received
0620 19:14:52.611814 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.611816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.611825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.611851 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:52.612115 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.612912 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.614385 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.614978 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.615029 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.615038 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.615267 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.615316 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:14:52.615901 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.611429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33907 (local address 127.19.66.193:46761)
0620 19:14:52.611656 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.611666 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:52.611744 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:52.611960 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:52.611961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:52.611963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:52.612039 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.612041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.613011 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.613012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.614283 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.614286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.615117 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.615118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.615186 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.615190 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:52.615714 (+ 524us) client_negotiation.cc:240] Negotiation successful
0620 19:14:52.615794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":102,"threads_started":1}
I20260620 19:14:52.616405 32747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94499ff5f1814c8d868765d61734d02c" candidate_uuid: "d788449969e1498abba96140b0fdedb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0b2e3e21dd44fd806065383647036b" is_pre_election: true
I20260620 19:14:52.616586 32747 raft_consensus.cc:2454] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d788449969e1498abba96140b0fdedb7 in term 0.
I20260620 19:14:52.616956 32549 leader_election.cc:302] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0b2e3e21dd44fd806065383647036b, d788449969e1498abba96140b0fdedb7; no voters:
I20260620 19:14:52.617093 499 raft_consensus.cc:2781] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:52.617136 499 raft_consensus.cc:482] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:52.617156 499 raft_consensus.cc:3037] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:52.618068 499 raft_consensus.cc:504] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.618201 499 leader_election.cc:288] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [CANDIDATE]: Term 1 election: Requested vote from peers 8c0b2e3e21dd44fd806065383647036b (127.19.66.194:33907), 9efeffef35924f1b821abf1cfa783f75 (127.19.66.195:43349)
I20260620 19:14:52.618593 32747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94499ff5f1814c8d868765d61734d02c" candidate_uuid: "d788449969e1498abba96140b0fdedb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c0b2e3e21dd44fd806065383647036b"
I20260620 19:14:52.618670 32747 raft_consensus.cc:3037] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:52.619474 32747 raft_consensus.cc:2454] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d788449969e1498abba96140b0fdedb7 in term 1.
I20260620 19:14:52.619710 32549 leader_election.cc:302] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c0b2e3e21dd44fd806065383647036b, d788449969e1498abba96140b0fdedb7; no voters:
I20260620 19:14:52.619818 499 raft_consensus.cc:2781] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:52.619949 499 raft_consensus.cc:686] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [term 1 LEADER]: Becoming Leader. State: Replica: d788449969e1498abba96140b0fdedb7, State: Running, Role: LEADER
I20260620 19:14:52.620038 499 consensus_queue.cc:237] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } }
I20260620 19:14:52.620882 32461 catalog_manager.cc:5495] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d788449969e1498abba96140b0fdedb7 (127.19.66.193). New cstate: current_term: 1 leader_uuid: "d788449969e1498abba96140b0fdedb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d788449969e1498abba96140b0fdedb7" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 45915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 } health_report { overall_health: UNKNOWN } } }
W20260620 19:14:52.622794 460 tablet.cc:2369] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:52.625180 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.611798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:51343 (local address 127.19.66.195:43349)
0620 19:14:52.611911 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.611913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.621944 (+ 10031us) server_negotiation.cc:407] Connection header received
0620 19:14:52.622041 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.622042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.622051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.622077 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:52.622244 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.622964 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.624237 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.624804 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.624872 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.624882 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.625070 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.625112 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260620 19:14:52.625519 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.611640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:43349 (local address 127.19.66.193:51343)
0620 19:14:52.621890 (+ 10250us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.621901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:52.621972 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:52.622140 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:52.622141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:52.622142 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:52.622179 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.622181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.623068 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.623069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.624137 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:52.624140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.624935 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.624936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:52.624996 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.624999 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:14:52.625411 (+ 412us) client_negotiation.cc:240] Negotiation successful
0620 19:14:52.625453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10237,"thread_start_us":53,"threads_started":1}
I20260620 19:14:52.626050 413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94499ff5f1814c8d868765d61734d02c" candidate_uuid: "d788449969e1498abba96140b0fdedb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efeffef35924f1b821abf1cfa783f75" is_pre_election: true
I20260620 19:14:52.626200 413 raft_consensus.cc:2454] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d788449969e1498abba96140b0fdedb7 in term 0.
I20260620 19:14:52.626327 412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94499ff5f1814c8d868765d61734d02c" candidate_uuid: "d788449969e1498abba96140b0fdedb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efeffef35924f1b821abf1cfa783f75"
I20260620 19:14:52.626375 412 raft_consensus.cc:3037] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:52.627274 412 raft_consensus.cc:2454] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d788449969e1498abba96140b0fdedb7 in term 1.
W20260620 19:14:52.652629 326 tablet.cc:2369] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:52.653996 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.650423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.19.66.193:45915)
0620 19:14:52.650460 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:52.650462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:52.650468 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:14:52.650507 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:52.650508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:52.650529 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:52.650596 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:52.650758 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.651455 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.652970 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:52.653254 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:52.653311 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:52.653505 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:52.653646 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:52.653710 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:52.653905 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:14:52.653926 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:14:52.656189 412 raft_consensus.cc:1261] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 1 FOLLOWER]: Refusing update from remote peer d788449969e1498abba96140b0fdedb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:52.656555 506 consensus_queue.cc:1035] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efeffef35924f1b821abf1cfa783f75" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:52.658732 32747 raft_consensus.cc:1261] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 1 FOLLOWER]: Refusing update from remote peer d788449969e1498abba96140b0fdedb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:52.659070 506 consensus_queue.cc:1035] T 94499ff5f1814c8d868765d61734d02c P d788449969e1498abba96140b0fdedb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c0b2e3e21dd44fd806065383647036b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:52.662707 509 mvcc.cc:205] Tried to move back new op lower bound from 7299001928316841984 to 7299001928172187648. Current Snapshot: MvccSnapshot[applied={T|T < 7299001928316841984}]
I20260620 19:14:52.663266 512 mvcc.cc:205] Tried to move back new op lower bound from 7299001928316841984 to 7299001928172187648. Current Snapshot: MvccSnapshot[applied={T|T < 7299001928316841984}]
I20260620 19:14:52.665848 510 mvcc.cc:205] Tried to move back new op lower bound from 7299001928316841984 to 7299001928172187648. Current Snapshot: MvccSnapshot[applied={T|T < 7299001928316841984}]
I20260620 19:14:52.667294 32461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:52.667383 32461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:52.668143 32461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a07624a33ac4036a176e90f4a35b728: Sending DeleteTablet for 3 replicas of tablet 94499ff5f1814c8d868765d61734d02c
I20260620 19:14:52.668437 32575 tablet_service.cc:1514] Processing DeleteTablet for tablet 94499ff5f1814c8d868765d61734d02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:52 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55580
I20260620 19:14:52.668926 32709 tablet_service.cc:1514] Processing DeleteTablet for tablet 94499ff5f1814c8d868765d61734d02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:52 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:45020
I20260620 19:14:52.669144 393 tablet_service.cc:1514] Processing DeleteTablet for tablet 94499ff5f1814c8d868765d61734d02c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:52 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:40454
I20260620 19:14:52.669729 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32500
I20260620 19:14:52.671589 519 tablet_replica.cc:330] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: stopping tablet replica
I20260620 19:14:52.671721 520 tablet_replica.cc:330] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: stopping tablet replica
I20260620 19:14:52.671937 520 raft_consensus.cc:2229] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:52.672062 519 raft_consensus.cc:2229] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:52.672221 519 raft_consensus.cc:2258] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:52.672333 520 raft_consensus.cc:2258] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:52.672968 519 ts_tablet_manager.cc:1905] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:52.674598 519 ts_tablet_manager.cc:1918] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:52.674729 519 log.cc:1196] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/94499ff5f1814c8d868765d61734d02c
I20260620 19:14:52.675036 519 ts_tablet_manager.cc:1939] T 94499ff5f1814c8d868765d61734d02c P 8c0b2e3e21dd44fd806065383647036b: Deleting consensus metadata
I20260620 19:14:52.675493 32448 catalog_manager.cc:4841] TS 8c0b2e3e21dd44fd806065383647036b (127.19.66.194:33907): tablet 94499ff5f1814c8d868765d61734d02c (table test-table [id=783ade3703da43c69736ea9dcd0adf28]) successfully deleted
I20260620 19:14:52.678993 520 ts_tablet_manager.cc:1905] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:52.680541 520 ts_tablet_manager.cc:1918] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:52.680688 520 log.cc:1196] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/94499ff5f1814c8d868765d61734d02c
I20260620 19:14:52.681126 520 ts_tablet_manager.cc:1939] T 94499ff5f1814c8d868765d61734d02c P 9efeffef35924f1b821abf1cfa783f75: Deleting consensus metadata
I20260620 19:14:52.681617 32448 catalog_manager.cc:4841] TS 9efeffef35924f1b821abf1cfa783f75 (127.19.66.195:43349): tablet 94499ff5f1814c8d868765d61734d02c (table test-table [id=783ade3703da43c69736ea9dcd0adf28]) successfully deleted
W20260620 19:14:52.682907 32446 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:45915 (error 108)
I20260620 19:14:52.683799 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:52.683688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:45915 (local address 127.0.0.1:55592)
0620 19:14:52.683736 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:52.683765 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:45915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:14:52.684003 32446 catalog_manager.cc:4584] TS d788449969e1498abba96140b0fdedb7 (127.19.66.193:45915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94499ff5f1814c8d868765d61734d02c: Network error: Client connection negotiation failed: client connection to 127.19.66.193:45915: connect: Connection refused (error 111)
I20260620 19:14:52.684234 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32663
I20260620 19:14:52.691242 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 327
I20260620 19:14:52.697433 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 32428
I20260620 19:14:52.702639 19723 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 19:14:53.036629 19723 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 19:14:53.036715 19723 mini_ranger.cc:66] Stopping Ranger...
I20260620 19:14:53.370690 19723 mini_ranger.cc:68] Stopped Ranger
2026-06-20 19:14:53.370 UTC [31495] LOG: received smart shutdown request
2026-06-20 19:14:53.374 UTC [31495] LOG: background worker "logical replication launcher" (PID 31504) exited with exit code 1
2026-06-20 19:14:53.374 UTC [31499] LOG: shutting down
2026-06-20 19:14:53.499 UTC [31495] LOG: database system is shut down
2026-06-20T19:14:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (44026 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:53 dist-test-slave-nq8r krb5kdc[530](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:53 dist-test-slave-nq8r krb5kdc[530](info): set up 2 sockets
Jun 20 19:14:53 dist-test-slave-nq8r krb5kdc[530](info): commencing operation
krb5kdc: starting...
W20260620 19:14:54.528520 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:14:54.632946 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43317
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:54.710197 543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:54.710412 543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:54.710443 543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:54.711562 543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:54.711603 543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:54.711620 543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:54.711637 543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:54.711652 543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:54.713176 543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:54.713414 543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:54.713610 543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:54.717964 549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:54.718035 551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:54.717972 548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:54.718202 543 server_base.cc:1034] running on GCE node
I20260620 19:14:54.718400 543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:54.718641 543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:54.719770 543 hybrid_clock.cc:648] HybridClock initialized: now 1781982894719757 us; error 27 us; skew 500 ppm
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:54.724720 543 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:54.725188 543 webserver.cc:460] Webserver started at http://127.19.66.254:39769/ using document root <none> and password file <none>
I20260620 19:14:54.725360 543 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:54.725399 543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:54.725487 543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:54.726445 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "a5a3c941695d4d3ba366b30aa9dd5c60"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "c7872a2f27f18d38f104c64a08491f91"
server_key_iv: "c7d79174be9037d371604273744e7c6d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.726722 543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "a5a3c941695d4d3ba366b30aa9dd5c60"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "c7872a2f27f18d38f104c64a08491f91"
server_key_iv: "c7d79174be9037d371604273744e7c6d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.727838 543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:14:54.728459 558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:54.728605 543 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:14:54.728682 543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "a5a3c941695d4d3ba366b30aa9dd5c60"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "c7872a2f27f18d38f104c64a08491f91"
server_key_iv: "c7d79174be9037d371604273744e7c6d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.728739 543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:54.746744 543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:54.748296 543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:54.748418 543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:54.752115 543 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:43317
I20260620 19:14:54.752148 610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:43317 every 8 connection(s)
I20260620 19:14:54.752442 543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:54.753242 611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:54.754675 611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:54.756819 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 543
I20260620 19:14:54.756938 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:14:54.757117 19723 external_mini_cluster.cc:1428] Setting key edad00050ddba712db2eec60226335bb
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.253@KRBTEST.COM; defaulting to no policy
I20260620 19:14:54.763639 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.755281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43317 (local address 127.0.0.1:43840)
0620 19:14:54.755438 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.755477 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:54.755932 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:54.756192 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:54.756194 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:54.756434 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:54.756580 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.756585 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.757701 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.757702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.758562 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.758565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.759055 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.759056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.759114 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.759554 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:54.759561 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:54.760871 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:54.762388 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.762393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.762395 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.762572 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.762764 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.762764 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.762765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.762840 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.763135 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:54.763140 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:54.763198 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 19:14:54.763276 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 19:14:54.763513 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":107,"thread_start_us":76,"threads_started":1}
I20260620 19:14:54.763640 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.755310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.19.66.254:43317)
0620 19:14:54.755447 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:54.755456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:54.755556 (+ 100us) server_negotiation.cc:407] Connection header received
0620 19:14:54.756081 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:54.756083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:54.756099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:54.756179 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:54.756691 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.757617 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.758650 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.758975 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.759083 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.760953 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:54.760960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:54.760960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:54.760967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:54.762314 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.762637 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.762640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.762641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.762710 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.762890 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.762891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.762891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.763078 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:54.763146 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:54.763305 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 19:14:54.763409 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260620 19:14:54.764381 611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:54.764674 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.764574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43506)
0620 19:14:54.764612 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.764653 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:14:54.764798 561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
I20260620 19:14:54.765214 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.765123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43510)
0620 19:14:54.765174 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.765197 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 19:14:54.765296 611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:42443: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
I20260620 19:14:54.776815 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:42443
--webserver_interface=127.19.66.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:54.814527 611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } attempt: 1
I20260620 19:14:54.815011 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.814882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43514)
0620 19:14:54.814942 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.814984 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 19:14:54.815315 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.815236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43520)
0620 19:14:54.815275 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.815298 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 19:14:54.815407 611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:42443: Network error: Client connection negotiation failed: client connection to 127.19.66.253:42443: connect: Connection refused (error 111)
W20260620 19:14:54.854125 618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:54.854310 618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:54.854329 618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:54.855365 618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:54.855396 618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:54.855409 618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:54.855420 618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:54.855434 618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:54.856868 618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:42443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.66.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:54.857056 618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:54.857225 618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:54.861495 623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:54.861526 626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:54.861624 618 server_base.cc:1034] running on GCE node
W20260620 19:14:54.861498 624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:54.861950 618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:54.862215 618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:54.863373 618 hybrid_clock.cc:648] HybridClock initialized: now 1781982894863355 us; error 27 us; skew 500 ppm
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:54.865586 618 init.cc:377] Logged in from keytab as kudu/127.19.66.253@KRBTEST.COM (short username kudu)
I20260620 19:14:54.865996 618 webserver.cc:460] Webserver started at http://127.19.66.253:43789/ using document root <none> and password file <none>
I20260620 19:14:54.866211 618 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:54.866258 618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:54.866380 618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:54.867347 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data/instance:
uuid: "7bd38e11adce4418a51aab827d8b2259"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "d61b837fa01ccd176f55fd4d332dfb28"
server_key_iv: "8e670e065e31a8a4c7c46d8c29754a8d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.867645 618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal/instance:
uuid: "7bd38e11adce4418a51aab827d8b2259"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "d61b837fa01ccd176f55fd4d332dfb28"
server_key_iv: "8e670e065e31a8a4c7c46d8c29754a8d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.868793 618 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:14:54.869443 633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:54.869583 618 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:54.869660 618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "7bd38e11adce4418a51aab827d8b2259"
format_stamp: "Formatted at 2026-06-20 19:14:54 on dist-test-slave-nq8r"
server_key: "d61b837fa01ccd176f55fd4d332dfb28"
server_key_iv: "8e670e065e31a8a4c7c46d8c29754a8d"
server_key_version: "encryptionkey@0"
I20260620 19:14:54.869719 618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:54.878343 618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:54.879863 618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:54.879982 618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:54.883565 618 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.253:42443
I20260620 19:14:54.883544 611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } attempt: 2
I20260620 19:14:54.883613 685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.253:42443 every 8 connection(s)
I20260620 19:14:54.883936 618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
I20260620 19:14:54.884585 686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:54.886428 686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:14:54.890802 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 618
I20260620 19:14:54.890903 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-1/wal/instance
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:54.891072 19723 external_mini_cluster.cc:1428] Setting key fc31a9558a36e73d457fd7671907d102
I20260620 19:14:54.893610 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.883794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43532)
0620 19:14:54.883840 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.883851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:54.883898 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:54.885235 (+ 1337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:54.885235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:54.885364 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:54.885416 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.885418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.886906 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.886907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.887509 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.887511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.888112 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.888113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.888194 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.888606 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:54.888610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:54.889705 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:54.891505 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.891505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.891506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.891739 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.892355 (+ 616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.892356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.892356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.892426 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.893261 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:54.893262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:54.893350 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 19:14:54.893422 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:14:54.893517 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:14:54.893812 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.883941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.19.66.253:42443)
0620 19:14:54.884443 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:54.884452 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:54.884482 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:14:54.885027 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:54.885035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:54.885104 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:54.885247 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:54.885504 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.886827 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.887578 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.888052 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.888128 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.889778 (+ 1650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:54.889785 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:54.889785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:54.889797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:54.891450 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.891806 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.891810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.891810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.892247 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.892892 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.892892 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.892893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.893149 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:54.893256 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:54.893497 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 19:14:54.893609 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":414,"threads_started":1}
I20260620 19:14:54.894205 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.887025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43317 (local address 127.0.0.1:43846)
0620 19:14:54.887151 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.887179 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:54.887224 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:54.887350 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:54.887353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:54.887630 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:54.887700 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.887702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.888351 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.888351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.889063 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.889066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.889408 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.889409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.889483 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.889894 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:54.889899 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:54.891396 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:54.892871 (+ 1475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.892878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.892880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.893117 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.893328 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.893329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.893329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.893383 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.893639 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:54.893645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:54.893690 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:14:54.893998 (+ 308us) client_negotiation.cc:240] Negotiation successful
0620 19:14:54.894113 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260620 19:14:54.894207 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.887195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.19.66.254:43317)
0620 19:14:54.887254 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:54.887256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:54.887264 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:54.887281 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:54.887282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:54.887291 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:54.887335 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:54.887769 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.888265 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.889143 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.889342 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.889393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.891480 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:54.891482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:54.891482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:54.891489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:54.892781 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.893212 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.893213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.893213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.893271 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.893450 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.893450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.893451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.893562 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:54.893598 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:54.894033 (+ 435us) server_negotiation.cc:299] Negotiation successful
0620 19:14:54.894111 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 19:14:54.894399 611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:54.894709 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.894609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:40982)
0620 19:14:54.894647 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.894682 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 19:14:54.894997 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.894926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:40990)
0620 19:14:54.894958 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.894978 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:14:54.895097 611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
I20260620 19:14:54.895066 686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:14:54 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.252@KRBTEST.COM" created.
I20260620 19:14:54.900908 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.895318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43538 (local address 127.19.66.253:42443)
0620 19:14:54.895346 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:54.895348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:54.895355 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:14:54.895399 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:54.895400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:54.895407 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:54.895448 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:54.895701 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.896150 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.897041 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.897222 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.897266 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.898834 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:54.898835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:54.898835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:54.898841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:54.899997 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.900266 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.900267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.900267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.900312 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:54.900434 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:54.900435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:54.900435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:54.900547 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:54.900600 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:54.900728 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:14:54.900805 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260620 19:14:54.901007 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.895281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43538)
0620 19:14:54.895318 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.895326 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:54.895357 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:54.895455 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:54.895456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:54.895592 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:54.895630 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.895631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.896214 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.896216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.896976 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:54.896979 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:54.897283 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:54.897284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:54.897342 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:54.897642 (+ 300us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:54.897646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:54.898765 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:54.900064 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.900065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.900066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.900210 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.900356 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:54.900356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:54.900357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:54.900398 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:54.900602 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:54.900603 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:54.900648 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:14:54.900708 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:14:54.900919 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":81}
I20260620 19:14:54.902081 686 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:54.902384 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.902271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41006)
0620 19:14:54.902311 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.902363 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:14:54.902519 637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
I20260620 19:14:54.902917 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.902829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41010)
0620 19:14:54.902870 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.902897 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 19:14:54.903010 686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
I20260620 19:14:54.913195 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:43085
--webserver_interface=127.19.66.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:54.938242 611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 1
I20260620 19:14:54.938700 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.938555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41018)
0620 19:14:54.938618 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.938664 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:14:54.939013 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.938929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41030)
0620 19:14:54.938967 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.938993 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 19:14:54.939107 611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
I20260620 19:14:54.952230 686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 1
I20260620 19:14:54.952657 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.952534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41038)
0620 19:14:54.952587 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.952626 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:14:54.952975 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.952902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41048)
0620 19:14:54.952930 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.952956 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 19:14:54.953069 686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
I20260620 19:14:54.986223 611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 2
I20260620 19:14:54.986654 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.986533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41062)
0620 19:14:54.986580 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.986611 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 19:14:54.987059 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:54.986972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41076)
0620 19:14:54.986999 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:54.987035 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 19:14:54.987186 611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
W20260620 19:14:54.998401 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:54.998596 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:54.998616 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:54.999750 693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:54.999787 693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:54.999800 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:54.999811 693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:54.999827 693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:55.001332 693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43317,127.19.66.253:42443,127.19.66.252:43085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:43085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.66.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:55.001524 693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:55.001699 693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:55.006070 699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:55.006102 698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:55.006084 701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:55.006389 693 server_base.cc:1034] running on GCE node
I20260620 19:14:55.006650 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:55.006913 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:55.008061 693 hybrid_clock.cc:648] HybridClock initialized: now 1781982895008049 us; error 27 us; skew 500 ppm
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:55.010387 693 init.cc:377] Logged in from keytab as kudu/127.19.66.252@KRBTEST.COM (short username kudu)
I20260620 19:14:55.010816 693 webserver.cc:460] Webserver started at http://127.19.66.252:35627/ using document root <none> and password file <none>
I20260620 19:14:55.011014 693 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:55.011062 693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:55.011180 693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:55.012178 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data/instance:
uuid: "311b6d41ebe34af6aafe35e92276ea51"
format_stamp: "Formatted at 2026-06-20 19:14:55 on dist-test-slave-nq8r"
server_key: "5ba8273cf96e4d0c396988ffbcbc68e2"
server_key_iv: "bea83483dd64157153cca50315a0b3b9"
server_key_version: "encryptionkey@0"
I20260620 19:14:55.012501 693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal/instance:
uuid: "311b6d41ebe34af6aafe35e92276ea51"
format_stamp: "Formatted at 2026-06-20 19:14:55 on dist-test-slave-nq8r"
server_key: "5ba8273cf96e4d0c396988ffbcbc68e2"
server_key_iv: "bea83483dd64157153cca50315a0b3b9"
server_key_version: "encryptionkey@0"
I20260620 19:14:55.013681 693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:14:55.014339 708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:55.014524 693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:14:55.014614 693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "311b6d41ebe34af6aafe35e92276ea51"
format_stamp: "Formatted at 2026-06-20 19:14:55 on dist-test-slave-nq8r"
server_key: "5ba8273cf96e4d0c396988ffbcbc68e2"
server_key_iv: "bea83483dd64157153cca50315a0b3b9"
server_key_version: "encryptionkey@0"
I20260620 19:14:55.014672 693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:55.021206 686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 2
I20260620 19:14:55.021566 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.021434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41078)
0620 19:14:55.021487 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.021529 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:14:55.021896 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.021808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41080)
0620 19:14:55.021848 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.021873 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:14:55.021974 686 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:43085: Network error: Client connection negotiation failed: client connection to 127.19.66.252:43085: connect: Connection refused (error 111)
I20260620 19:14:55.025347 693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:55.027269 693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:55.027392 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:55.030972 693 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.252:43085
I20260620 19:14:55.031008 760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.252:43085 every 8 connection(s)
I20260620 19:14:55.031337 693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
I20260620 19:14:55.032147 761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:55.033634 761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:55.037177 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 693
I20260620 19:14:55.037266 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781982751812665-19723-0/minicluster-data/master-2/wal/instance
I20260620 19:14:55.037446 19723 external_mini_cluster.cc:1428] Setting key 71820d16d34467261343a2d5969642c8
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:55.041927 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.034223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.19.66.254:43317)
0620 19:14:55.034268 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.034270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.034434 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:14:55.034858 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.034859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.034867 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.034917 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.035426 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.035878 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.036754 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.036953 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.036998 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.039051 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.039052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.039053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.039059 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.040696 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.041094 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.041096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.041096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.041151 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.041357 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.041358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.041358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.041463 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.041518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.041758 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.041840 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:14:55.042070 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.034190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:43317 (local address 127.0.0.1:43854)
0620 19:14:55.034327 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.034365 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:55.034790 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:55.034968 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:55.034971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:55.035187 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:55.035367 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.035371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.035933 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.035933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.036699 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.036701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.037012 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.037013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.037114 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.037567 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:55.037574 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:55.038960 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:55.040770 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.040774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.040776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.041007 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.041203 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.041203 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.041204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.041302 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.041516 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:55.041522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:55.041644 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 19:14:55.041729 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:14:55.041862 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":55,"threads_started":1}
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:14:55.042557 761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:55.045301 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.037776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.19.66.252:43085)
0620 19:14:55.038016 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.038019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.038034 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:14:55.038050 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.038052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.038066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.038122 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.038677 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.039429 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.040360 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.040643 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.040693 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.042753 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.042759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.042760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.042767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.044260 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.044591 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.044593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.044594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.044637 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.044790 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.044791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.044791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.044962 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.045099 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.045142 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.045213 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":74,"threads_started":1}
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:14:55.049710 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.042781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:42443 (local address 127.0.0.1:43552)
0620 19:14:55.042877 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.042887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:55.042927 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:55.043071 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:55.043072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:55.043246 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:55.043287 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.043289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.044113 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.044114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.044750 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.044752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.045167 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.045168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.045239 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.045581 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:55.045585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:55.046723 (+ 1138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:55.048677 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.048678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.048678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.048823 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.049081 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.049082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.049082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.049178 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.049457 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:55.049458 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:55.049489 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:14:55.049541 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 19:14:55.049624 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 19:14:55.049750 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.042846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.19.66.253:42443)
0620 19:14:55.042883 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.042885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.042915 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:14:55.043000 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.043002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.043012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.043058 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.043354 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.044043 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.044816 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.045100 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.045159 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.046793 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.046794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.046795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.046800 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.048611 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.048921 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.048922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.048922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.048978 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.049274 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.049276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.049276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.049386 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.049421 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.049582 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.049671 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:55.050179 761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:14:55.052573 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.045975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.19.66.254:43317)
0620 19:14:55.046012 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.046014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.046022 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:55.046061 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.046062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.046070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.046129 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.046503 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.047056 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.047896 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.048176 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.048225 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.050305 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.050306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.050306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.050311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.051665 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.051936 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.051937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.051937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.051986 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.052142 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.052142 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.052143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.052242 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.052396 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.052429 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.052492 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982895, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:14:55.056099 763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.050376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41096)
0620 19:14:55.050401 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.050410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:55.050443 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:55.050554 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:55.050555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:55.050685 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:55.050717 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.050719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.051544 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.051545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.051985 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.051986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.052308 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.052308 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.052352 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.052809 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:55.052814 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:55.053863 (+ 1049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:55.055160 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.055160 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.055161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.055345 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.055512 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.055512 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.055513 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.055569 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.055773 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:55.055774 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:55.055816 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:14:55.055888 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:14:55.055976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260620 19:14:55.056150 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.050406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.19.66.252:43085)
0620 19:14:55.050459 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.050461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.050482 (+ 21us) server_negotiation.cc:407] Connection header received
0620 19:14:55.050498 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.050499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.050509 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.050549 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.050782 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.051474 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.052055 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.052255 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.052305 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.053930 (+ 1625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.053931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.053931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.053937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.055079 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.055409 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.055410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.055410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.055456 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.055639 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.055639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.055640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.055722 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.055765 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.055903 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.056015 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":33}
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:14:55.057592 761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: Bootstrap starting.
I20260620 19:14:55.058588 761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:55.058872 761 log.cc:824] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:55.059581 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.053109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43554 (local address 127.19.66.253:42443)
0620 19:14:55.053147 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.053149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.053157 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:55.053194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.053195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.053203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.053253 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.053642 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.054085 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.054674 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.054865 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.054913 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.057097 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.057098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.057099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.057105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.058536 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.058848 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.058849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.058849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.058904 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.059086 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.059087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.059087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.059225 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.059382 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.059467 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.059522 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:14:55.059677 761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: No bootstrap required, opened a new log
I20260620 19:14:55.061000 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:55.061416 761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.061532 761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:55.061563 761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311b6d41ebe34af6aafe35e92276ea51, State: Initialized, Role: FOLLOWER
I20260620 19:14:55.061681 761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.061977 761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:55.061937 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.062047 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:55.064026 693 master_runner.cc:186] Error getting master registration for 127.19.66.254:43317: OK, instance_id { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" instance_seqno: 1781982894751127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:55.066737 693 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:14:55.066915 785 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:55.066964 785 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:55.069854 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.062276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.19.66.253:42443)
0620 19:14:55.062312 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.062314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.062766 (+ 452us) server_negotiation.cc:407] Connection header received
0620 19:14:55.062796 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.062797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.062806 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.062859 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.063390 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.064052 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.065115 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.065409 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.065447 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.067005 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.067006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.067007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.067013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.068698 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.069082 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.069083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.069084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.069140 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.069316 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.069317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.069317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.069423 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.069493 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.069685 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.069760 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:14:55.069854 614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.062296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.19.66.254:43317)
0620 19:14:55.062333 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.062335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.062487 (+ 152us) server_negotiation.cc:407] Connection header received
0620 19:14:55.062509 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.062510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.062520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.062560 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.063224 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.064109 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.065114 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.065409 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.065446 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.067036 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.067037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.067038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.067043 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.068563 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.069039 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.069040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.069041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.069095 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.069284 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.069285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.069285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.069399 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.069481 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.069698 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.069766 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:14:55.071566 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.062568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.19.66.252:43085)
0620 19:14:55.062598 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.062600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.063618 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 19:14:55.063707 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.063708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.063716 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.063741 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.064429 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.065122 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.066491 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.066749 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.066791 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.068963 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.068964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.068965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.068972 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.070483 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.070863 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.070864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.070865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.070927 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.071087 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.071087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.071088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.071219 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.071298 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.071417 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.071485 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:14:55.094321 611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 3
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:14:55.100992 613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.094615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41118)
0620 19:14:55.094682 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.094696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:55.094740 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:55.094878 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:55.094879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:55.095018 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:55.095061 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.095062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.095920 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.095921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.096390 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.096391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.096801 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.096801 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.096834 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.097316 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:55.097320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:55.098643 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:55.100195 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.100196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.100196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.100340 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.100502 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.100502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.100502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.100543 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.100743 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:55.100744 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:55.100777 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 19:14:55.100820 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:14:55.100926 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:14:55.101033 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.094715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.19.66.252:43085)
0620 19:14:55.094776 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.094778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.094788 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:14:55.094809 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.094810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.094819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.094876 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.095116 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.095857 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.096448 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.096741 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.096800 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.098727 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.098729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.098729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.098736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.100112 (+ 1376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.100395 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.100396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.100396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.100451 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.100578 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.100579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.100579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.100697 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.100755 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.100864 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.100937 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:14:55.102675 611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: Bootstrap starting.
I20260620 19:14:55.103403 611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:55.103694 611 log.cc:824] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:55.104488 611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: No bootstrap required, opened a new log
I20260620 19:14:55.106457 611 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.106603 611 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:55.106649 611 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a3c941695d4d3ba366b30aa9dd5c60, State: Initialized, Role: FOLLOWER
I20260620 19:14:55.106768 611 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.107009 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.107082 611 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:55.107081 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:14:55.108832 799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:55.108948 799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:55.109594 543 master_runner.cc:186] Error getting master registration for 127.19.66.253:42443: OK, instance_id { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" instance_seqno: 1781982894882581 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:14:55.112337 543 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:55.113097 686 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } attempt: 3
Jun 20 19:14:55 dist-test-slave-nq8r krb5kdc[530](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982894, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:14:55.120963 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.113342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:43085 (local address 127.0.0.1:41120)
0620 19:14:55.113389 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:55.113399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:55.113432 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:55.113685 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:55.113686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:55.113857 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:55.113902 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.113903 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.114768 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.114769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.115564 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:55.115567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.116011 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.116012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:55.116079 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.116539 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:55.116543 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:55.118120 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:55.119862 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.119863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.119864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.120085 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.120321 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:55.120322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:55.120322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:55.120385 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:55.120645 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:55.120646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:55.120691 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:14:55.120735 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:14:55.120831 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:14:55.121508 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:55.113515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.19.66.252:43085)
0620 19:14:55.113568 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:55.113570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:55.113579 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:14:55.113599 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:55.113600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:55.113609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:55.113669 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:55.113966 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.114691 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.115645 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:55.115938 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:55.116009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:55.118219 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:55.118220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:55.118221 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:55.118228 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:55.119779 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.120158 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.120159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.120159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.120237 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:55.120451 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:55.120452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:55.120452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:55.120577 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:55.121297 (+ 720us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:55.121343 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:14:55.121416 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:14:55.123165 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: Bootstrap starting.
I20260620 19:14:55.123878 686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:55.124158 686 log.cc:824] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:55.124989 686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: No bootstrap required, opened a new log
I20260620 19:14:55.126909 686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.127063 686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:55.127103 686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bd38e11adce4418a51aab827d8b2259, State: Initialized, Role: FOLLOWER
I20260620 19:14:55.127215 686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.127513 686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 19:14:55.129313 813 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:14:55.129407 813 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:14:55.127475 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.129570 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:14:55.130474 639 tablet.cc:2369] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:55.131422 618 master_runner.cc:263] No leader master found from master 7bd38e11adce4418a51aab827d8b2259
I20260620 19:14:55.131464 618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:14:55.142074 768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:55.142195 768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.142414 768 leader_election.cc:288] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a3c941695d4d3ba366b30aa9dd5c60 (127.19.66.254:43317), 7bd38e11adce4418a51aab827d8b2259 (127.19.66.253:42443)
I20260620 19:14:55.143366 660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "311b6d41ebe34af6aafe35e92276ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd38e11adce4418a51aab827d8b2259" is_pre_election: true
I20260620 19:14:55.143522 660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311b6d41ebe34af6aafe35e92276ea51 in term 0.
I20260620 19:14:55.144182 711 leader_election.cc:302] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311b6d41ebe34af6aafe35e92276ea51, 7bd38e11adce4418a51aab827d8b2259; no voters:
I20260620 19:14:55.144742 585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "311b6d41ebe34af6aafe35e92276ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" is_pre_election: true
I20260620 19:14:55.144965 768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:55.145025 768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:55.145046 768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:55.145923 768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.146039 768 leader_election.cc:288] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [CANDIDATE]: Term 1 election: Requested vote from peers a5a3c941695d4d3ba366b30aa9dd5c60 (127.19.66.254:43317), 7bd38e11adce4418a51aab827d8b2259 (127.19.66.253:42443)
I20260620 19:14:55.146706 585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311b6d41ebe34af6aafe35e92276ea51 in term 0.
I20260620 19:14:55.146265 584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "311b6d41ebe34af6aafe35e92276ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60"
I20260620 19:14:55.146804 584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:55.147620 584 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311b6d41ebe34af6aafe35e92276ea51 in term 1.
I20260620 19:14:55.147868 710 leader_election.cc:302] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311b6d41ebe34af6aafe35e92276ea51, a5a3c941695d4d3ba366b30aa9dd5c60; no voters:
I20260620 19:14:55.148015 660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "311b6d41ebe34af6aafe35e92276ea51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bd38e11adce4418a51aab827d8b2259"
I20260620 19:14:55.148084 660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:55.148957 660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311b6d41ebe34af6aafe35e92276ea51 in term 1.
I20260620 19:14:55.149206 768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:55.149540 768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [term 1 LEADER]: Becoming Leader. State: Replica: 311b6d41ebe34af6aafe35e92276ea51, State: Running, Role: LEADER
I20260620 19:14:55.149633 768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } }
I20260620 19:14:55.149992 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311b6d41ebe34af6aafe35e92276ea51. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.150071 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:55.150353 817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:55.150568 817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:55.152362 660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [term 1 FOLLOWER]: Refusing update from remote peer 311b6d41ebe34af6aafe35e92276ea51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:55.152689 768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:55.153925 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311b6d41ebe34af6aafe35e92276ea51. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.154033 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:55.154206 584 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [term 1 FOLLOWER]: Refusing update from remote peer 311b6d41ebe34af6aafe35e92276ea51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:55.154505 768 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:55.155612 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 311b6d41ebe34af6aafe35e92276ea51. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.155694 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:55.156685 818 mvcc.cc:205] Tried to move back new op lower bound from 7299001938542125056 to 7299001938533494784. Current Snapshot: MvccSnapshot[applied={T|T < 7299001938542125056}]
I20260620 19:14:55.156971 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.157043 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:55.157437 788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.157509 788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:55.158178 821 mvcc.cc:205] Tried to move back new op lower bound from 7299001938542125056 to 7299001938533494784. Current Snapshot: MvccSnapshot[applied={T|T < 7299001938542125056}]
I20260620 19:14:55.159742 768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.159816 768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:55.160169 802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "311b6d41ebe34af6aafe35e92276ea51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a3c941695d4d3ba366b30aa9dd5c60" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43317 } } peers { permanent_uuid: "7bd38e11adce4418a51aab827d8b2259" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 42443 } } peers { permanent_uuid: "311b6d41ebe34af6aafe35e92276ea51" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 43085 } } }
I20260620 19:14:55.160254 802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:14:55.160789 819 mvcc.cc:205] Tried to move back new op lower bound from 7299001938542125056 to 7299001938533494784. Current Snapshot: MvccSnapshot[applied={T|T < 7299001938542125056}]
I20260620 19:14:55.162165 817 catalog_manager.cc:1347] Generated new cluster ID: 161b359a0112459f938400690d2d0198
I20260620 19:14:55.162207 817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:55.182469 817 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:55.183259 817 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:55.191653 817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 311b6d41ebe34af6aafe35e92276ea51: Generated new TSK 0
I20260620 19:14:55.191920 817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:56.109397 799 catalog_manager.cc:1259] Loaded cluster ID: 161b359a0112459f938400690d2d0198
I20260620 19:14:56.109470 799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: loading cluster ID for follower catalog manager: success
I20260620 19:14:56.110286 799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: acquiring CA information for follower catalog manager: success
I20260620 19:14:56.110565 799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a5a3c941695d4d3ba366b30aa9dd5c60: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:56.129843 813 catalog_manager.cc:1259] Loaded cluster ID: 161b359a0112459f938400690d2d0198
I20260620 19:14:56.129911 813 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: loading cluster ID for follower catalog manager: success
I20260620 19:14:56.130724 813 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: acquiring CA information for follower catalog manager: success
I20260620 19:14:56.131006 813 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7bd38e11adce4418a51aab827d8b2259: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:14:56.251153 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 543
I20260620 19:14:56.255892 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 618
I20260620 19:14:56.260011 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 693
2026-06-20T19:14:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2678 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:56 dist-test-slave-nq8r krb5kdc[834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:56 dist-test-slave-nq8r krb5kdc[834](info): set up 2 sockets
Jun 20 19:14:56 dist-test-slave-nq8r krb5kdc[834](info): commencing operation
krb5kdc: starting...
W20260620 19:14:57.196676 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:14:57.303190 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40031
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:57.379856 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:57.380049 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:57.380079 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:57.381196 847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:57.381235 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:57.381253 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:57.381270 847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:57.381284 847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:57.382697 847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:57.382920 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:57.383108 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:57.387511 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:57.387537 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:57.387521 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:57.387802 847 server_base.cc:1034] running on GCE node
I20260620 19:14:57.388007 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:57.388289 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:57.389426 847 hybrid_clock.cc:648] HybridClock initialized: now 1781982897389408 us; error 28 us; skew 500 ppm
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982897, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:57.391637 847 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:57.392023 847 webserver.cc:460] Webserver started at http://127.19.66.254:46819/ using document root <none> and password file <none>
I20260620 19:14:57.392232 847 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:57.392279 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:57.392401 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:57.393456 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "31296bf6db4241e69344f6ce714bf47f"
format_stamp: "Formatted at 2026-06-20 19:14:57 on dist-test-slave-nq8r"
server_key: "a43620a328bd3ab5446e0c5b33178e45"
server_key_iv: "047befce64b573374cce87bef76d25cf"
server_key_version: "encryptionkey@0"
I20260620 19:14:57.393770 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "31296bf6db4241e69344f6ce714bf47f"
format_stamp: "Formatted at 2026-06-20 19:14:57 on dist-test-slave-nq8r"
server_key: "a43620a328bd3ab5446e0c5b33178e45"
server_key_iv: "047befce64b573374cce87bef76d25cf"
server_key_version: "encryptionkey@0"
I20260620 19:14:57.394932 847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:14:57.395587 862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:57.395725 847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:14:57.395802 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "31296bf6db4241e69344f6ce714bf47f"
format_stamp: "Formatted at 2026-06-20 19:14:57 on dist-test-slave-nq8r"
server_key: "a43620a328bd3ab5446e0c5b33178e45"
server_key_iv: "047befce64b573374cce87bef76d25cf"
server_key_version: "encryptionkey@0"
I20260620 19:14:57.395859 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:57.409533 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:57.411285 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:57.411417 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:57.415009 847 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:40031
I20260620 19:14:57.415045 914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:40031 every 8 connection(s)
I20260620 19:14:57.415374 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:57.416206 915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:57.417084 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 847
I20260620 19:14:57.417162 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:14:57.417336 19723 external_mini_cluster.cc:1428] Setting key 8e1c0a890297109f6e442671193da46f
I20260620 19:14:57.419284 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f: Bootstrap starting.
I20260620 19:14:57.419852 915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:57.420111 915 log.cc:824] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:57.420708 915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f: No bootstrap required, opened a new log
I20260620 19:14:57.422628 915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } }
I20260620 19:14:57.422761 915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:57.422796 915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31296bf6db4241e69344f6ce714bf47f, State: Initialized, Role: FOLLOWER
I20260620 19:14:57.422951 915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } }
I20260620 19:14:57.423043 915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:57.423074 915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:57.423120 915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 0 FOLLOWER]: Advancing to term 1
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[834](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:57.423738 915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } }
I20260620 19:14:57.423839 915 leader_election.cc:302] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31296bf6db4241e69344f6ce714bf47f; no voters:
I20260620 19:14:57.424021 915 leader_election.cc:288] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:57.424065 920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:57.424229 920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [term 1 LEADER]: Becoming Leader. State: Replica: 31296bf6db4241e69344f6ce714bf47f, State: Running, Role: LEADER
I20260620 19:14:57.424265 915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:57.424367 920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } }
I20260620 19:14:57.424685 920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31296bf6db4241e69344f6ce714bf47f. Latest consensus state: current_term: 1 leader_uuid: "31296bf6db4241e69344f6ce714bf47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } } }
I20260620 19:14:57.424695 921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31296bf6db4241e69344f6ce714bf47f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31296bf6db4241e69344f6ce714bf47f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40031 } } }
I20260620 19:14:57.424875 921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:57.424865 920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:57.425642 925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:57.426129 925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:57.427675 925 catalog_manager.cc:1347] Generated new cluster ID: e5b9c762dc7d448eb6493491986c3512
I20260620 19:14:57.427699 918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:57.417798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.19.66.254:40031)
0620 19:14:57.417942 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:57.417959 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:57.417988 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:14:57.418410 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:57.418416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:57.418463 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:57.418576 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:57.419314 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:57.420238 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:57.421030 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:57.421456 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:57.421550 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:57.423913 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:57.423917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:57.423917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:57.423930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:57.425923 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:57.426302 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:57.426305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:57.426306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:57.426494 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:57.426809 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:57.426809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:57.426810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:57.427052 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:57.427241 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:57.427376 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:14:57.427509 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260620 19:14:57.427721 925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:57.428561 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:14:57.455439 925 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:57.456188 925 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:57.460479 925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31296bf6db4241e69344f6ce714bf47f: Generated new TSK 0
I20260620 19:14:57.460656 925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:14:57.482203 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 847
2026-06-20T19:14:57Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1221 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[942](info): set up 2 sockets
Jun 20 19:14:57 dist-test-slave-nq8r krb5kdc[942](info): commencing operation
krb5kdc: starting...
W20260620 19:14:58.424804 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:14:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:14:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:14:58.529278 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:58.607602 955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:58.607812 955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:58.607842 955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:58.609102 955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:58.609148 955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:58.609167 955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:58.609184 955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:58.609200 955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:58.610674 955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:58.610903 955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:58.611104 955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:58.615607 961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:58.615573 960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:58.615649 963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:58.615756 955 server_base.cc:1034] running on GCE node
I20260620 19:14:58.616007 955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:58.616313 955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:58.617463 955 hybrid_clock.cc:648] HybridClock initialized: now 1781982898617443 us; error 32 us; skew 500 ppm
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:58.619805 955 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:58.620257 955 webserver.cc:460] Webserver started at http://127.19.66.254:44553/ using document root <none> and password file <none>
I20260620 19:14:58.620462 955 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:58.620509 955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:58.620630 955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:58.621678 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "66169377a3c1401fa3215d6f104a63d0"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "803694d7af537060e8a08d45f7c34522"
server_key_iv: "8b2f498ff216cac429b3348466529e22"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.622009 955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "66169377a3c1401fa3215d6f104a63d0"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "803694d7af537060e8a08d45f7c34522"
server_key_iv: "8b2f498ff216cac429b3348466529e22"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.623184 955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:14:58.623849 970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.624032 955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:14:58.624118 955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "66169377a3c1401fa3215d6f104a63d0"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "803694d7af537060e8a08d45f7c34522"
server_key_iv: "8b2f498ff216cac429b3348466529e22"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.624183 955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:58.642133 955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:58.644007 955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:58.644145 955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:58.647929 955 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:36181
I20260620 19:14:58.647962 1022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:36181 every 8 connection(s)
I20260620 19:14:58.648291 955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:58.649197 1023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:58.651355 1023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap starting.
I20260620 19:14:58.651856 1023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:58.652069 1023 log.cc:824] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:58.652681 1023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: No bootstrap required, opened a new log
I20260620 19:14:58.653261 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 955
I20260620 19:14:58.653363 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:14:58.653537 19723 external_mini_cluster.cc:1428] Setting key aa1cbefd85795a4ac28aa76fdde96f08
I20260620 19:14:58.654466 1023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:58.654609 1023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:58.654642 1023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Initialized, Role: FOLLOWER
I20260620 19:14:58.654796 1023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:58.654875 1023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:58.654912 1023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:58.654958 1023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:58.655857 1023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:58.655970 1023 leader_election.cc:302] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66169377a3c1401fa3215d6f104a63d0; no voters:
I20260620 19:14:58.656170 1023 leader_election.cc:288] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:14:58.656225 1028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:58.656356 1028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 LEADER]: Becoming Leader. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Running, Role: LEADER
I20260620 19:14:58.656458 1023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:58.656484 1028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:58.656836 1028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66169377a3c1401fa3215d6f104a63d0. Latest consensus state: current_term: 1 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:14:58.656986 1028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:58.657222 1037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:58.657775 1037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:58.658615 1030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:14:58.658689 1030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:58.659490 1037 catalog_manager.cc:1347] Generated new cluster ID: e010c0fa463f4d90ab4f67f28b3cabca
I20260620 19:14:58.659538 1037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:58.664598 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:58.653995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35094 (local address 127.19.66.254:36181)
0620 19:14:58.654171 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:58.654179 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:58.654204 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:14:58.654441 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:58.654447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:58.654490 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:58.654614 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:58.655341 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.656387 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.657426 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.658006 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.658115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:58.660799 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:58.660803 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:58.660803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:58.660815 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:58.662714 (+ 1899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.663110 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.663113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.663113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.663225 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.663409 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.663410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.663410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.664006 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:58.664151 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:58.664266 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 19:14:58.664394 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260620 19:14:58.667452 1037 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:14:58.668128 1037 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:58.676013 1037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Generated new TSK 0
I20260620 19:14:58.676188 1037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:14:58.700990 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:58.791298 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:58.791510 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:58.791534 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:58.792673 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:58.792714 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:58.792738 1049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:14:58.794194 1049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:58.794410 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:58.794627 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:58.799172 1054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:58.799180 1057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:58.799300 1055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:58.799546 1049 server_base.cc:1034] running on GCE node
I20260620 19:14:58.799757 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:58.799968 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:58.801105 1049 hybrid_clock.cc:648] HybridClock initialized: now 1781982898801092 us; error 28 us; skew 500 ppm
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:58.803393 1049 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:14:58.803778 1049 webserver.cc:460] Webserver started at http://127.19.66.193:41413/ using document root <none> and password file <none>
I20260620 19:14:58.803946 1049 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:58.803984 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:58.804069 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:58.805075 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "3f59b25a8d4e4276816672f463838be1"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "e8c629f92cb3007dd060ec86830ed254"
server_key_iv: "6bb6969f69beac675135334f8e4f0c9d"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.805362 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "3f59b25a8d4e4276816672f463838be1"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "e8c629f92cb3007dd060ec86830ed254"
server_key_iv: "6bb6969f69beac675135334f8e4f0c9d"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.806501 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:58.807119 1064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.807307 1049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:14:58.807384 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "3f59b25a8d4e4276816672f463838be1"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "e8c629f92cb3007dd060ec86830ed254"
server_key_iv: "6bb6969f69beac675135334f8e4f0c9d"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.807436 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:58.829376 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:58.831188 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:58.831292 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:58.831476 1049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:58.831759 1049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:58.831792 1049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.831815 1049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:58.831838 1049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.837137 1049 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:46863
I20260620 19:14:58.837208 1177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:46863 every 8 connection(s)
I20260620 19:14:58.837481 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:58.845921 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1049
I20260620 19:14:58.846020 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:14:58.846220 19723 external_mini_cluster.cc:1428] Setting key c2ec03d306992a57fa4ac6aca924f87e
I20260620 19:14:58.846259 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:58.838144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:60735 (local address 127.19.66.254:36181)
0620 19:14:58.838190 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:58.838192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:58.838373 (+ 181us) server_negotiation.cc:407] Connection header received
0620 19:14:58.839078 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:58.839079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:58.839092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:58.839138 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:58.840009 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.840493 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.841519 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.841730 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.841779 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:58.843762 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:58.843764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:58.843764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:58.843772 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:58.845080 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.845442 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.845442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.845442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.845490 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.845688 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.845688 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.845688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.845796 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:58.845853 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:58.846071 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 19:14:58.846174 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:14:58.846419 1180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:58.838116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:60735)
0620 19:14:58.838272 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:58.838307 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:58.838956 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:58.839195 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:58.839198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:58.839755 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:58.839942 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:58.839946 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.840560 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.840561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:58.841454 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:58.841457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.841800 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.841801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:58.841905 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:58.842353 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:58.842361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:58.843665 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:58.845149 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:58.845156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:58.845158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:58.845382 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:58.845551 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:58.845551 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:58.845552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:58.845646 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:58.845852 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:58.845855 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:58.845961 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:14:58.846046 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 19:14:58.846201 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260620 19:14:58.846905 1178 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:58.846993 1178 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:58.847167 1178 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:58.847635 986 ts_manager.cc:194] Registered new tserver with Master: 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863)
I20260620 19:14:58.848477 986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:60735
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:14:58.867348 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:58.944977 1183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:58.945169 1183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:58.945190 1183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:58.946308 1183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:58.946346 1183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:58.946372 1183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:14:58.947774 1183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:58.947979 1183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:58.948179 1183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:58.952626 1188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:58.952710 1183 server_base.cc:1034] running on GCE node
W20260620 19:14:58.952687 1189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:58.952626 1191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:58.953053 1183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:58.953276 1183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:58.954401 1183 hybrid_clock.cc:648] HybridClock initialized: now 1781982898954383 us; error 33 us; skew 500 ppm
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:58.956727 1183 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:14:58.957175 1183 webserver.cc:460] Webserver started at http://127.19.66.194:43491/ using document root <none> and password file <none>
I20260620 19:14:58.957377 1183 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:58.957422 1183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:58.957547 1183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:58.958555 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "6b56617c010241fc80faac600e13dbed"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "4f57f7fc339b42263e02ba9fb2744c07"
server_key_iv: "a1c95fc51ff3b81c3b4c696fd7c0a8ea"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.958873 1183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "6b56617c010241fc80faac600e13dbed"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "4f57f7fc339b42263e02ba9fb2744c07"
server_key_iv: "a1c95fc51ff3b81c3b4c696fd7c0a8ea"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.960016 1183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:14:58.960647 1198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.960778 1183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:14:58.960872 1183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "6b56617c010241fc80faac600e13dbed"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "4f57f7fc339b42263e02ba9fb2744c07"
server_key_iv: "a1c95fc51ff3b81c3b4c696fd7c0a8ea"
server_key_version: "encryptionkey@0"
I20260620 19:14:58.960938 1183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:58.967669 1183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:58.969647 1183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:58.969769 1183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:58.969942 1183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:58.970198 1183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:58.970228 1183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.970254 1183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:58.970278 1183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:58.976006 1183 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38955
I20260620 19:14:58.976078 1311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38955 every 8 connection(s)
I20260620 19:14:58.976383 1183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:58.981380 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1183
I20260620 19:14:58.981474 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:14:58.981648 19723 external_mini_cluster.cc:1428] Setting key 657dddd619b1680c142890b5985e662d
Jun 20 19:14:58 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982898, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:58.986249 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:58.977395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:46805 (local address 127.19.66.254:36181)
0620 19:14:58.977449 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:58.977451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:58.977462 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:14:58.978062 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:58.978064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:58.978092 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:58.978151 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:58.979274 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.979869 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.981270 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.981572 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.981645 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:58.983790 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:58.983791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:58.983791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:58.983814 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:58.985041 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.985393 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.985394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.985394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.985450 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:58.985665 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:58.985666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:58.985667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:58.985776 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:58.985831 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:58.986079 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 19:14:58.986163 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:14:58.986420 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:58.977100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:46805)
0620 19:14:58.977276 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:58.977321 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:58.977906 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:58.978234 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:58.978238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:58.978934 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:58.979188 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:58.979194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.979947 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.979948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:58.981187 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:58.981191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:58.981662 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:58.981663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:58.981810 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:58.982265 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:58.982274 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:58.983708 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:58.985101 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:58.985109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:58.985111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:58.985328 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:58.985501 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:58.985502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:58.985502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:58.985605 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:58.985828 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:58.985834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:58.985962 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:14:58.986053 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 19:14:58.986212 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260620 19:14:58.986825 1312 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:58.986912 1312 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:58.987074 1312 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:58.987430 986 ts_manager.cc:194] Registered new tserver with Master: 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
I20260620 19:14:58.987920 986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:46805
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:14:59.003013 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:59.079728 1317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:59.079926 1317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:59.079948 1317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:59.081040 1317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:59.081075 1317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:59.081115 1317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:14:59.082473 1317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:59.082661 1317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:59.082831 1317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:59.087455 1322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.087452 1323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.087455 1325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.087670 1317 server_base.cc:1034] running on GCE node
I20260620 19:14:59.087916 1317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:59.088150 1317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:59.089315 1317 hybrid_clock.cc:648] HybridClock initialized: now 1781982899089300 us; error 28 us; skew 500 ppm
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:59.091779 1317 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:14:59.092221 1317 webserver.cc:460] Webserver started at http://127.19.66.195:44015/ using document root <none> and password file <none>
I20260620 19:14:59.092417 1317 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:59.092465 1317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:59.092590 1317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:14:59.093653 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "1341f9f81264423484314fe2c8236bca"
format_stamp: "Formatted at 2026-06-20 19:14:59 on dist-test-slave-nq8r"
server_key: "44f3477fd79da707dcf117ddb001122b"
server_key_iv: "07555beb0c586e7d252ef13b32b4569f"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.093977 1317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "1341f9f81264423484314fe2c8236bca"
format_stamp: "Formatted at 2026-06-20 19:14:59 on dist-test-slave-nq8r"
server_key: "44f3477fd79da707dcf117ddb001122b"
server_key_iv: "07555beb0c586e7d252ef13b32b4569f"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.095156 1317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:14:59.095803 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.095947 1317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:14:59.096030 1317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "1341f9f81264423484314fe2c8236bca"
format_stamp: "Formatted at 2026-06-20 19:14:59 on dist-test-slave-nq8r"
server_key: "44f3477fd79da707dcf117ddb001122b"
server_key_iv: "07555beb0c586e7d252ef13b32b4569f"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.096097 1317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:59.119707 1317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:59.121968 1317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:59.122109 1317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:59.122339 1317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:59.122658 1317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:59.122690 1317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.122720 1317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:59.122736 1317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.128427 1317 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:38647
I20260620 19:14:59.128468 1445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:38647 every 8 connection(s)
I20260620 19:14:59.128763 1317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:59.137130 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.129424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:47827 (local address 127.19.66.254:36181)
0620 19:14:59.129476 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.129478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.129671 (+ 193us) server_negotiation.cc:407] Connection header received
0620 19:14:59.130183 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.130184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.130196 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.130249 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.131178 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.131675 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.132616 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.132814 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.132885 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.134806 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.134808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.134808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.134815 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.135893 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.136235 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.136236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.136237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.136287 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.136498 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.136499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.136499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.136606 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.136661 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.136933 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.137017 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:14:59.137229 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1317
I20260620 19:14:59.137266 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.129406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:47827)
0620 19:14:59.129564 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.129598 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.130080 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.130320 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.130331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.130894 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:59.131081 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.131086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.131753 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.131754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.132537 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.132539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.132902 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.132903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.133016 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.133438 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:59.133446 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:59.134719 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:59.135961 (+ 1242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.135967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.135969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.136174 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.136338 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.136339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.136339 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.136445 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.136666 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:59.136671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:59.136799 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.136913 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.137059 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260620 19:14:59.137316 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:14:59.137511 19723 external_mini_cluster.cc:1428] Setting key 6ed96d55fdb78d2df6db3df79a2b3801
I20260620 19:14:59.137665 1446 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:59.137759 1446 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:59.137940 1446 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:59.138343 986 ts_manager.cc:194] Registered new tserver with Master: 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647)
I20260620 19:14:59.138686 986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:47827
I20260620 19:14:59.149070 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:59.155267 1027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.150358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.19.66.254:36181)
0620 19:14:59.150411 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.150413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.150497 (+ 84us) server_negotiation.cc:407] Connection header received
0620 19:14:59.150543 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.150544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.150552 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.150596 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.151067 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.151534 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.152204 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.152374 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.152416 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.153437 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.153438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.153439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.153445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.154461 (+ 1016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.154733 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.154733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.154734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.154775 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.154884 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.154884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.154884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.155006 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.155074 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.155140 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.155207 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:14:59.156620 986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:59.158295 986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:59.163763 1458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:38647 (local address 127.0.0.1:44898)
0620 19:14:59.160664 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.160673 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.160719 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.160911 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.160912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.160913 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.160942 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.160943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.161741 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.161742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.162610 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.162612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.163471 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.163547 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.163561 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.163640 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.163688 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":50,"thread_start_us":33,"threads_started":1}
I20260620 19:14:59.163776 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.19.66.195:38647)
0620 19:14:59.160624 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.160628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.160766 (+ 138us) server_negotiation.cc:407] Connection header received
0620 19:14:59.160804 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.160806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.160820 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.160905 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.160993 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.161649 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.162701 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.163387 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163452 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.163506 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.163664 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.163700 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":84,"threads_started":1}
I20260620 19:14:59.164157 1454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:38955 (local address 127.0.0.1:41784)
0620 19:14:59.160282 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.160321 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.160379 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.160659 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.160662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.160665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.160734 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.160736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.161709 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.161710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.163132 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.163134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.163938 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.164003 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.164007 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.164055 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.164093 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":32,"thread_start_us":64,"threads_started":1}
I20260620 19:14:59.164289 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41784 (local address 127.19.66.194:38955)
0620 19:14:59.160418 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.160422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.160430 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:59.160468 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.160470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.160483 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.160559 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.160797 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.161618 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163203 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.163858 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163928 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.163977 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.164073 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.164216 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260620 19:14:59.164732 1380 tablet_service.cc:1467] Processing CreateTablet for tablet 567f73099a9940c68321dc97f8b4c60a (DEFAULT_TABLE table=test-table [id=370c5ee231944d5285db88a09eee0ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.165097 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.0.0.1:35482)
0620 19:14:59.160467 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.160474 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.160522 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.160687 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.160687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.160689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.160737 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.160738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.162887 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.162888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.163880 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.163883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.164872 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.164873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.164934 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.164937 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.164996 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.165032 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":2,"spinlock_wait_cycles":896,"thread_start_us":64,"threads_started":1}
I20260620 19:14:59.165102 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f73099a9940c68321dc97f8b4c60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.165122 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.160189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.19.66.193:46863)
0620 19:14:59.160394 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.160399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.160536 (+ 137us) server_negotiation.cc:407] Connection header received
0620 19:14:59.160591 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.160593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.160607 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.160676 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.161733 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.162788 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.163966 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.164782 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.164843 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.164915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.165009 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.165049 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":100,"threads_started":1}
I20260620 19:14:59.165136 1246 tablet_service.cc:1467] Processing CreateTablet for tablet 567f73099a9940c68321dc97f8b4c60a (DEFAULT_TABLE table=test-table [id=370c5ee231944d5285db88a09eee0ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.165431 1246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f73099a9940c68321dc97f8b4c60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.168311 1460 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Bootstrap starting.
I20260620 19:14:59.168665 1461 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Bootstrap starting.
I20260620 19:14:59.169004 1460 tablet_bootstrap.cc:654] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.169080 1112 tablet_service.cc:1467] Processing CreateTablet for tablet 567f73099a9940c68321dc97f8b4c60a (DEFAULT_TABLE table=test-table [id=370c5ee231944d5285db88a09eee0ad4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.169281 1461 tablet_bootstrap.cc:654] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.169376 1112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567f73099a9940c68321dc97f8b4c60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.169620 1460 log.cc:824] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.169818 1461 log.cc:824] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.170356 1460 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: No bootstrap required, opened a new log
I20260620 19:14:59.170415 1460 ts_tablet_manager.cc:1397] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:14:59.170549 1461 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: No bootstrap required, opened a new log
I20260620 19:14:59.170621 1461 ts_tablet_manager.cc:1397] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:59.171873 1460 raft_consensus.cc:348] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.172012 1460 raft_consensus.cc:374] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.172044 1460 raft_consensus.cc:729] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b56617c010241fc80faac600e13dbed, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.172155 1460 consensus_queue.cc:260] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.172423 1460 ts_tablet_manager.cc:1428] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:14:59.172463 1312 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
I20260620 19:14:59.172480 1461 raft_consensus.cc:348] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.172626 1461 raft_consensus.cc:374] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.172673 1464 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Bootstrap starting.
I20260620 19:14:59.172714 1461 raft_consensus.cc:729] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1341f9f81264423484314fe2c8236bca, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.172829 1461 consensus_queue.cc:260] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.173121 1446 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
I20260620 19:14:59.173345 1464 tablet_bootstrap.cc:654] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.173374 1461 ts_tablet_manager.cc:1428] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:14:59.173879 1464 log.cc:824] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.174510 1464 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: No bootstrap required, opened a new log
I20260620 19:14:59.174561 1464 ts_tablet_manager.cc:1397] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:59.175980 1464 raft_consensus.cc:348] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.176100 1464 raft_consensus.cc:374] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.176119 1464 raft_consensus.cc:729] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f59b25a8d4e4276816672f463838be1, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.176223 1464 consensus_queue.cc:260] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.176445 1464 ts_tablet_manager.cc:1428] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:14:59.176502 1178 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
I20260620 19:14:59.205183 1466 raft_consensus.cc:482] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:59.205299 1466 raft_consensus.cc:504] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.205644 1466 leader_election.cc:288] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
I20260620 19:14:59.208990 1457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.205878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:39207 (local address 127.19.66.194:38955)
0620 19:14:59.205919 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.205920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.206023 (+ 103us) server_negotiation.cc:407] Connection header received
0620 19:14:59.206065 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.206066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.206074 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.206112 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.206240 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.206913 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208262 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.208662 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208719 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.208727 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.208883 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.208925 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:14:59.209018 1456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.205837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:48423 (local address 127.19.66.193:46863)
0620 19:14:59.205883 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.205885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.205895 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:14:59.205930 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.205931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.205941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.205996 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.206155 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.206874 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208283 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.208688 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208757 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.208765 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.208920 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.208961 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:14:59.209028 1469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.205787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:38955 (local address 127.19.66.195:39207)
0620 19:14:59.205960 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.205968 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.206016 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.206127 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.206127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.206129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.206184 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.206186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.206968 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.206968 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.208185 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.208187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.208735 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208737 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.208805 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.208810 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.208871 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.208915 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":15,"thread_start_us":108,"threads_started":1}
I20260620 19:14:59.209033 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.205787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.19.66.195:48423)
0620 19:14:59.205838 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.205848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.205885 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.206015 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.206015 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.206017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.206095 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.206097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.206942 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.206943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.208185 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.208187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.208773 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.208774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.208840 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.208864 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.208915 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.208953 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":35}
I20260620 19:14:59.209380 1265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f73099a9940c68321dc97f8b4c60a" candidate_uuid: "1341f9f81264423484314fe2c8236bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b56617c010241fc80faac600e13dbed" is_pre_election: true
I20260620 19:14:59.209406 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f73099a9940c68321dc97f8b4c60a" candidate_uuid: "1341f9f81264423484314fe2c8236bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f59b25a8d4e4276816672f463838be1" is_pre_election: true
I20260620 19:14:59.209529 1265 raft_consensus.cc:2454] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1341f9f81264423484314fe2c8236bca in term 0.
I20260620 19:14:59.209537 1132 raft_consensus.cc:2454] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1341f9f81264423484314fe2c8236bca in term 0.
I20260620 19:14:59.209740 1335 leader_election.cc:302] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed; no voters:
I20260620 19:14:59.209900 1466 raft_consensus.cc:2781] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:59.209934 1466 raft_consensus.cc:482] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:59.209949 1466 raft_consensus.cc:3037] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.210632 1466 raft_consensus.cc:504] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.210737 1466 leader_election.cc:288] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [CANDIDATE]: Term 1 election: Requested vote from peers 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
I20260620 19:14:59.210888 1132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f73099a9940c68321dc97f8b4c60a" candidate_uuid: "1341f9f81264423484314fe2c8236bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f59b25a8d4e4276816672f463838be1"
I20260620 19:14:59.210903 1265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "567f73099a9940c68321dc97f8b4c60a" candidate_uuid: "1341f9f81264423484314fe2c8236bca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b56617c010241fc80faac600e13dbed"
I20260620 19:14:59.210958 1132 raft_consensus.cc:3037] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.210961 1265 raft_consensus.cc:3037] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.211601 1265 raft_consensus.cc:2454] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1341f9f81264423484314fe2c8236bca in term 1.
I20260620 19:14:59.211632 1132 raft_consensus.cc:2454] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1341f9f81264423484314fe2c8236bca in term 1.
I20260620 19:14:59.211751 1335 leader_election.cc:302] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed; no voters:
I20260620 19:14:59.211839 1466 raft_consensus.cc:2781] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:59.211977 1466 raft_consensus.cc:686] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 1 LEADER]: Becoming Leader. State: Replica: 1341f9f81264423484314fe2c8236bca, State: Running, Role: LEADER
I20260620 19:14:59.212067 1466 consensus_queue.cc:237] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.212824 987 catalog_manager.cc:5495] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca reported cstate change: term changed from 0 to 1, leader changed from <none> to 1341f9f81264423484314fe2c8236bca (127.19.66.195). New cstate: current_term: 1 leader_uuid: "1341f9f81264423484314fe2c8236bca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20260620 19:14:59.223394 1459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.221531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.19.66.195:38647)
0620 19:14:59.221566 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.221568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.221583 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:14:59.221623 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.221623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.221637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.221699 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:59.221785 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.222213 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.222854 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.223021 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.223061 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.223169 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:59.223256 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:59.223307 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.223318 (+ 11us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.223333 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:14:59.224889 1132 raft_consensus.cc:1261] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Refusing update from remote peer 1341f9f81264423484314fe2c8236bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:59.224907 1265 raft_consensus.cc:1261] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Refusing update from remote peer 1341f9f81264423484314fe2c8236bca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:59.225185 1466 consensus_queue.cc:1035] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:59.225309 1470 consensus_queue.cc:1035] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:14:59.226960 1313 tablet.cc:2369] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:59.227566 1473 mvcc.cc:205] Tried to move back new op lower bound from 7299001955223105536 to 7299001955173289984. Current Snapshot: MvccSnapshot[applied={T|T < 7299001955223105536}]
I20260620 19:14:59.227764 1474 mvcc.cc:205] Tried to move back new op lower bound from 7299001955223105536 to 7299001955173289984. Current Snapshot: MvccSnapshot[applied={T|T < 7299001955223105536}]
I20260620 19:14:59.229732 986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:59.229810 986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:59.230466 986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Sending DeleteTablet for 3 replicas of tablet 567f73099a9940c68321dc97f8b4c60a
I20260620 19:14:59.230792 1112 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f73099a9940c68321dc97f8b4c60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:35482
I20260620 19:14:59.230819 1380 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f73099a9940c68321dc97f8b4c60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:44898
I20260620 19:14:59.230947 1246 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f73099a9940c68321dc97f8b4c60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:41784
I20260620 19:14:59.231062 1483 tablet_replica.cc:330] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: stopping tablet replica
I20260620 19:14:59.231055 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1049
I20260620 19:14:59.231150 1483 raft_consensus.cc:2229] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:59.231211 1485 tablet_replica.cc:330] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: stopping tablet replica
I20260620 19:14:59.231299 1483 raft_consensus.cc:2258] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.231334 1485 raft_consensus.cc:2229] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:59.231511 1485 raft_consensus.cc:2258] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.231662 1484 tablet_replica.cc:330] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: stopping tablet replica
I20260620 19:14:59.231746 1484 raft_consensus.cc:2229] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:14:59.231850 1484 raft_consensus.cc:2258] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.232194 1485 ts_tablet_manager.cc:1905] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:59.233536 1485 ts_tablet_manager.cc:1918] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:59.233584 1484 ts_tablet_manager.cc:1905] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:59.233649 1485 log.cc:1196] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/567f73099a9940c68321dc97f8b4c60a
I20260620 19:14:59.233949 1485 ts_tablet_manager.cc:1939] T 567f73099a9940c68321dc97f8b4c60a P 6b56617c010241fc80faac600e13dbed: Deleting consensus metadata
I20260620 19:14:59.234337 973 catalog_manager.cc:4841] TS 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): tablet 567f73099a9940c68321dc97f8b4c60a (table test-table [id=370c5ee231944d5285db88a09eee0ad4]) successfully deleted
I20260620 19:14:59.235086 1484 ts_tablet_manager.cc:1918] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:59.235145 1484 log.cc:1196] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/567f73099a9940c68321dc97f8b4c60a
I20260620 19:14:59.235379 1484 ts_tablet_manager.cc:1939] T 567f73099a9940c68321dc97f8b4c60a P 1341f9f81264423484314fe2c8236bca: Deleting consensus metadata
I20260620 19:14:59.235702 972 catalog_manager.cc:4841] TS 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): tablet 567f73099a9940c68321dc97f8b4c60a (table test-table [id=370c5ee231944d5285db88a09eee0ad4]) successfully deleted
W20260620 19:14:59.236137 972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:46863 (error 108)
I20260620 19:14:59.236544 1455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.236441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.0.0.1:35484)
0620 19:14:59.236484 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.236517 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:14:59.236653 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1183
W20260620 19:14:59.237480 972 catalog_manager.cc:4584] TS 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 567f73099a9940c68321dc97f8b4c60a: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46863: connect: Connection refused (error 111)
I20260620 19:14:59.242734 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1317
I20260620 19:14:59.246779 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 955
I20260620 19:14:59.251546 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--webserver_interface=127.19.66.254
--webserver_port=44553
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:14:59.329197 1487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:59.329386 1487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:59.329414 1487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:59.330515 1487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:14:59.330554 1487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:59.330580 1487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:59.330596 1487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:14:59.330613 1487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:14:59.332032 1487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=44553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:59.332259 1487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:59.332448 1487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:59.340183 1496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.340150 1493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.340178 1494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.340332 1487 server_base.cc:1034] running on GCE node
I20260620 19:14:59.340668 1487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:59.340981 1487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:59.342121 1487 hybrid_clock.cc:648] HybridClock initialized: now 1781982899342109 us; error 28 us; skew 500 ppm
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:59.344390 1487 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:14:59.344791 1487 webserver.cc:460] Webserver started at http://127.19.66.254:44553/ using document root <none> and password file <none>
I20260620 19:14:59.345019 1487 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:59.345081 1487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:59.346369 1487 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:59.346961 1503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.347100 1487 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.347196 1487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "66169377a3c1401fa3215d6f104a63d0"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "803694d7af537060e8a08d45f7c34522"
server_key_iv: "8b2f498ff216cac429b3348466529e22"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.347581 1487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:59.357620 1487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:59.359262 1487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:59.359382 1487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:59.363008 1487 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:36181
I20260620 19:14:59.363039 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:36181 every 8 connection(s)
I20260620 19:14:59.363396 1487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:14:59.364874 1556 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:14:59.365527 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap starting.
I20260620 19:14:59.365527 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1487
I20260620 19:14:59.365778 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:46863
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=41413
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:59.373706 1556 log.cc:824] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.375450 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:14:59.375715 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap complete.
I20260620 19:14:59.377781 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:59.378008 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.378157 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:59.378214 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:14:59.378263 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:14:59.378307 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:14:59.379288 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:59.379379 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66169377a3c1401fa3215d6f104a63d0; no voters:
I20260620 19:14:59.379530 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:14:59.379575 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:14:59.379679 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 LEADER]: Becoming Leader. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Running, Role: LEADER
I20260620 19:14:59.379782 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:14:59.379763 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:14:59.380038 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:14:59.380120 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:59.380330 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66169377a3c1401fa3215d6f104a63d0. Latest consensus state: current_term: 2 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:14:59.380424 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:14:59.380553 1565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:14:59.381332 1565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:14:59.381425 1565 catalog_manager.cc:1259] Loaded cluster ID: e010c0fa463f4d90ab4f67f28b3cabca
I20260620 19:14:59.381453 1565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:14:59.381762 1565 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 19:14:59.386400 1565 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 19:14:59.389145 1565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:14:59.389319 1565 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Loaded TSK: 0
I20260620 19:14:59.389515 1565 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:14:59.447870 1558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:59.448074 1558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:59.448096 1558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:59.449267 1558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:59.449306 1558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:59.449329 1558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:14:59.450920 1558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=41413
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:59.451161 1558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:59.451372 1558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:59.455894 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.455906 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.455977 1558 server_base.cc:1034] running on GCE node
W20260620 19:14:59.455926 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.456298 1558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:59.456508 1558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:59.457648 1558 hybrid_clock.cc:648] HybridClock initialized: now 1781982899457630 us; error 30 us; skew 500 ppm
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:59.460132 1558 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:14:59.460558 1558 webserver.cc:460] Webserver started at http://127.19.66.193:41413/ using document root <none> and password file <none>
I20260620 19:14:59.460754 1558 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:59.460815 1558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:59.462132 1558 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:59.462769 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.462914 1558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:14:59.462992 1558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "3f59b25a8d4e4276816672f463838be1"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "e8c629f92cb3007dd060ec86830ed254"
server_key_iv: "6bb6969f69beac675135334f8e4f0c9d"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.463310 1558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:59.503897 1558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:59.505715 1558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:59.505841 1558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:59.506086 1558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:59.506511 1600 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:14:59.507942 1558 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:14:59.507997 1558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:14:59.508033 1558 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:14:59.508670 1558 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:14:59.508701 1558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:59.508767 1600 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Bootstrap starting.
I20260620 19:14:59.515172 1558 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:46863
I20260620 19:14:59.515264 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:46863 every 8 connection(s)
I20260620 19:14:59.515610 1558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:14:59.517213 1600 log.cc:824] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.519379 1600 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:14:59.519652 1600 tablet_bootstrap.cc:492] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Bootstrap complete.
I20260620 19:14:59.520679 1600 ts_tablet_manager.cc:1397] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260620 19:14:59.521032 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1558
I20260620 19:14:59.521317 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:38955
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=43491
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:14:59.521775 1600 raft_consensus.cc:348] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.522084 1600 raft_consensus.cc:729] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f59b25a8d4e4276816672f463838be1, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.522239 1600 consensus_queue.cc:260] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } } peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } }
I20260620 19:14:59.522498 1600 ts_tablet_manager.cc:1428] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:59.527132 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.516398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:54695 (local address 127.19.66.254:36181)
0620 19:14:59.516590 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.516598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.516640 (+ 42us) server_negotiation.cc:407] Connection header received
0620 19:14:59.517223 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.517229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.517282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.517421 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.518226 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.519114 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.520540 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.521103 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.521220 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.523798 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.523802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.523802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.523814 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.525383 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.525851 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.525853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.525854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.525932 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.526230 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.526231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.526231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.526437 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.526495 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.526881 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.526993 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
I20260620 19:14:59.527165 1710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.516255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:54695)
0620 19:14:59.516424 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.516465 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.517008 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.517485 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.517488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.517813 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:59.518081 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.518089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.519220 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.519222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.520442 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.520446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.521340 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.521342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.521495 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.522068 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:59.522077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:59.523683 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:59.525458 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.525466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.525468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.525779 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.526012 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.526013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.526014 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.526169 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.526496 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:59.526502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:59.526675 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.526794 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.526966 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260620 19:14:59.527865 1708 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:59.527947 1708 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:59.528097 1708 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:59.528744 1520 ts_manager.cc:194] Registered new tserver with Master: 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863)
I20260620 19:14:59.529836 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:54695
I20260620 19:14:59.530711 1708 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.193@KRBTEST.COM
I20260620 19:14:59.537005 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.529301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.0.0.1:35494)
0620 19:14:59.529464 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.529506 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.529571 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.529768 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.529772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.530104 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:59.530163 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.530166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.531237 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.531238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.532160 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.532164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.532576 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.532577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.532648 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.533085 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:59.533091 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:59.534402 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:59.535957 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.535963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.535965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.536165 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.536341 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.536341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.536342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.536399 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.536636 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:59.536642 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:59.536691 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.536764 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.536905 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260620 19:14:59.537005 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.529486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.19.66.193:46863)
0620 19:14:59.529618 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.529623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.529631 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:59.529674 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.529676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.529688 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.529754 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.530249 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.531177 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.532243 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.532519 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.532566 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.534478 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.534485 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.534485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.534492 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.535882 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.536234 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.536237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.536237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.536288 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.536446 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.536447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.536447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.536577 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.536646 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.536802 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.536908 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260620 19:14:59.537348 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 567f73099a9940c68321dc97f8b4c60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:35494
I20260620 19:14:59.537811 1717 tablet_replica.cc:330] stopping tablet replica
I20260620 19:14:59.537921 1717 raft_consensus.cc:2229] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:59.537998 1717 raft_consensus.cc:2258] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.538302 1717 ts_tablet_manager.cc:1905] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:59.540400 1717 ts_tablet_manager.cc:1918] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:59.540472 1717 log.cc:1196] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/567f73099a9940c68321dc97f8b4c60a
I20260620 19:14:59.540704 1717 ts_tablet_manager.cc:1939] T 567f73099a9940c68321dc97f8b4c60a P 3f59b25a8d4e4276816672f463838be1: Deleting consensus metadata
I20260620 19:14:59.541231 1505 catalog_manager.cc:4841] TS 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863): tablet 567f73099a9940c68321dc97f8b4c60a (table test-table [id=370c5ee231944d5285db88a09eee0ad4]) successfully deleted
W20260620 19:14:59.603425 1714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:59.603610 1714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:59.603631 1714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:59.604741 1714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:59.604779 1714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:59.604805 1714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:14:59.606220 1714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=43491
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:59.606441 1714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:59.606642 1714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:59.611193 1725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.611255 1722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.611228 1723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.611464 1714 server_base.cc:1034] running on GCE node
I20260620 19:14:59.611637 1714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:59.611842 1714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:59.612969 1714 hybrid_clock.cc:648] HybridClock initialized: now 1781982899612959 us; error 31 us; skew 500 ppm
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:59.615207 1714 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:14:59.615597 1714 webserver.cc:460] Webserver started at http://127.19.66.194:43491/ using document root <none> and password file <none>
I20260620 19:14:59.615801 1714 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:59.615875 1714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:59.617152 1714 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:14:59.617638 1732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.617774 1714 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.617844 1714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "6b56617c010241fc80faac600e13dbed"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "4f57f7fc339b42263e02ba9fb2744c07"
server_key_iv: "a1c95fc51ff3b81c3b4c696fd7c0a8ea"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.618180 1714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:59.633072 1714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:59.634689 1714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:59.634814 1714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:59.635005 1714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:59.635283 1714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:59.635314 1714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.635336 1714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:59.635360 1714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.640944 1714 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38955
I20260620 19:14:59.640995 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38955 every 8 connection(s)
I20260620 19:14:59.641325 1714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:14:59.646354 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1714
I20260620 19:14:59.646638 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:38647
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=44015
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:59.650817 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.642045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:35471 (local address 127.19.66.254:36181)
0620 19:14:59.642108 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.642110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.642290 (+ 180us) server_negotiation.cc:407] Connection header received
0620 19:14:59.642788 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.642789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.642805 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.642857 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.643774 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.644285 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.645281 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.645502 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.645562 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.647911 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.647913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.647913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.647921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.649403 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.649857 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.649858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.649858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.649918 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.650176 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.650177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.650177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.650315 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.650375 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.650627 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.650722 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:14:59.650949 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.642011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:35471)
0620 19:14:59.642171 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.642207 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.642689 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.642924 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.642927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.643477 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:59.643677 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.643682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.644368 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.644369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.645199 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.645202 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.645581 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.645582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.645696 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.646133 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:59.646141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:59.647793 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:59.649487 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.649493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.649495 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.649788 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.649974 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.649974 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.649974 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.650104 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.650376 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:59.650382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:59.650502 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.650589 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.650734 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260620 19:14:59.651432 1846 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:59.651518 1846 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:59.651696 1846 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:59.652037 1520 ts_manager.cc:194] Registered new tserver with Master: 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
I20260620 19:14:59.652438 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:35471
W20260620 19:14:59.726712 1849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:14:59.726923 1849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:14:59.726954 1849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:14:59.728111 1849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:14:59.728153 1849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:14:59.728188 1849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:14:59.729642 1849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:38647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=44015
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:14:59.729888 1849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:14:59.730101 1849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:14:59.734760 1857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.734934 1849 server_base.cc:1034] running on GCE node
W20260620 19:14:59.734762 1854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:14:59.734762 1855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:14:59.735248 1849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:14:59.735481 1849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:14:59.736613 1849 hybrid_clock.cc:648] HybridClock initialized: now 1781982899736610 us; error 40 us; skew 500 ppm
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:14:59.743197 1849 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:14:59.743642 1849 webserver.cc:460] Webserver started at http://127.19.66.195:44015/ using document root <none> and password file <none>
I20260620 19:14:59.743862 1849 fs_manager.cc:359] Metadata directory not provided
I20260620 19:14:59.743922 1849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:14:59.745289 1849 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:59.745849 1864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.746013 1849 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.746097 1849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "1341f9f81264423484314fe2c8236bca"
format_stamp: "Formatted at 2026-06-20 19:14:59 on dist-test-slave-nq8r"
server_key: "44f3477fd79da707dcf117ddb001122b"
server_key_iv: "07555beb0c586e7d252ef13b32b4569f"
server_key_version: "encryptionkey@0"
I20260620 19:14:59.746419 1849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:14:59.763432 1849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:14:59.765551 1849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:14:59.765679 1849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:14:59.765846 1849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:14:59.766116 1849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:14:59.766148 1849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.766185 1849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:14:59.766209 1849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:14:59.771471 1849 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:38647
I20260620 19:14:59.771601 1977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:38647 every 8 connection(s)
I20260620 19:14:59.771840 1849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:14:59 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982899, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:14:59.779975 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.772483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:51585 (local address 127.19.66.254:36181)
0620 19:14:59.772534 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.772536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.772762 (+ 226us) server_negotiation.cc:407] Connection header received
0620 19:14:59.773242 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.773243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.773253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.773306 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.774193 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.774689 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.775607 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.775783 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.775826 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.777747 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.777748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.777748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.777755 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.778856 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.779254 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.779255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.779256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.779306 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.779491 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.779491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.779492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.779587 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.779660 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.779853 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.779912 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:14:59.780143 1980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.772482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:51585)
0620 19:14:59.772643 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.772682 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.773151 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.773364 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.773368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.773903 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:14:59.774101 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.774106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.774755 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.774756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.775544 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.775546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.775849 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.775850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.775956 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.776419 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:14:59.776427 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:14:59.777668 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:14:59.778919 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.778924 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.778925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.779118 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.779354 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:14:59.779354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:14:59.779355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:14:59.779450 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:14:59.779650 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:14:59.779653 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:14:59.779760 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.779834 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.779944 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 19:14:59.780606 1978 heartbeater.cc:347] Connected to a master server at 127.19.66.254:36181
I20260620 19:14:59.780689 1978 heartbeater.cc:464] Registering TS with master...
I20260620 19:14:59.780872 1978 heartbeater.cc:511] Master 127.19.66.254:36181 requested a full tablet report, sending...
I20260620 19:14:59.781240 1520 ts_manager.cc:194] Registered new tserver with Master: 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647)
I20260620 19:14:59.781585 1520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:51585
I20260620 19:14:59.781801 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 1849
I20260620 19:14:59.786835 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.782108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.19.66.254:36181)
0620 19:14:59.782144 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.782145 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.782273 (+ 128us) server_negotiation.cc:407] Connection header received
0620 19:14:59.782313 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.782314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.782323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.782376 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:14:59.782827 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.783258 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.783933 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.784099 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.784139 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.785108 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:14:59.785109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:14:59.785109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:14:59.785114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:14:59.786090 (+ 976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.786361 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.786362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.786362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.786401 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:14:59.786506 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:14:59.786506 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:14:59.786506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:14:59.786581 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:14:59.786705 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.786730 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.786780 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:14:59.787439 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:14:59.789863 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.787730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.19.66.254:36181)
0620 19:14:59.787785 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.787787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.787854 (+ 67us) server_negotiation.cc:407] Connection header received
0620 19:14:59.787908 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.787909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.787949 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.788001 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:59.788073 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.788509 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.789237 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.789414 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.789477 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.789603 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:59.789693 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:59.789772 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.789788 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.789803 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:14:59.790153 1520 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:14:59.792048 1520 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:14:59.794107 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 446752e22e2b4d519843c4dfc7f7785f (DEFAULT_TABLE table=test-table [id=3cc31b71a92143d997bbdb1f1bf027f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.794313 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446752e22e2b4d519843c4dfc7f7785f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.795792 1600 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Bootstrap starting.
I20260620 19:14:59.796121 1600 tablet_bootstrap.cc:654] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.796700 1600 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: No bootstrap required, opened a new log
I20260620 19:14:59.796748 1600 ts_tablet_manager.cc:1397] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 19:14:59.797122 1600 raft_consensus.cc:348] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.797196 1600 raft_consensus.cc:374] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.797243 1600 raft_consensus.cc:729] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f59b25a8d4e4276816672f463838be1, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.797286 1600 consensus_queue.cc:260] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.797379 1600 ts_tablet_manager.cc:1428] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:14:59.797689 1983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.793780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:38955 (local address 127.0.0.1:41790)
0620 19:14:59.793979 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.793988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.794033 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.794351 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.794352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.794354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.794396 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.794398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.795563 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.795563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.796552 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.796556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.797425 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.797426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.797494 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.797513 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.797561 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.797604 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":13,"thread_start_us":62,"threads_started":1}
I20260620 19:14:59.797701 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.793939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.19.66.194:38955)
0620 19:14:59.794187 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.794192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.794203 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:14:59.794240 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.794242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.794257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.794345 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.794463 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.795473 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.796642 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.797268 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.797333 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.797390 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.797585 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.797623 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":96,"threads_started":1}
I20260620 19:14:59.798576 1780 tablet_service.cc:1467] Processing CreateTablet for tablet 446752e22e2b4d519843c4dfc7f7785f (DEFAULT_TABLE table=test-table [id=3cc31b71a92143d997bbdb1f1bf027f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.798873 1780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446752e22e2b4d519843c4dfc7f7785f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.799278 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.793967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.19.66.195:38647)
0620 19:14:59.795860 (+ 1893us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.795865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.795875 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:14:59.795910 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.795913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.795932 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.796150 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.796216 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.797229 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.798413 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.798997 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.799056 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.799098 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.799194 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.799223 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1857,"thread_start_us":104,"threads_started":1}
I20260620 19:14:59.799292 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.793780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:38647 (local address 127.0.0.1:44914)
0620 19:14:59.793829 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.793840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.793884 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.796037 (+ 2153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.796038 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.796040 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.796081 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.796082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.797342 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.797343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.798339 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.798341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.799057 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.799058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.799119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.799123 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.799186 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.799230 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 19:14:59.799965 1912 tablet_service.cc:1467] Processing CreateTablet for tablet 446752e22e2b4d519843c4dfc7f7785f (DEFAULT_TABLE table=test-table [id=3cc31b71a92143d997bbdb1f1bf027f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:14:59.800282 1912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 446752e22e2b4d519843c4dfc7f7785f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:14:59.802196 1987 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Bootstrap starting.
I20260620 19:14:59.802736 1987 tablet_bootstrap.cc:654] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.803220 1987 log.cc:824] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.803261 1988 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Bootstrap starting.
I20260620 19:14:59.803812 1988 tablet_bootstrap.cc:654] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Neither blocks nor log segments found. Creating new log.
I20260620 19:14:59.803941 1987 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: No bootstrap required, opened a new log
I20260620 19:14:59.804009 1987 ts_tablet_manager.cc:1397] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:59.804347 1988 log.cc:824] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:14:59.805094 1988 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: No bootstrap required, opened a new log
I20260620 19:14:59.805161 1988 ts_tablet_manager.cc:1397] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:14:59.805863 1987 raft_consensus.cc:348] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.806028 1987 raft_consensus.cc:374] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.806082 1987 raft_consensus.cc:729] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b56617c010241fc80faac600e13dbed, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.806196 1987 consensus_queue.cc:260] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.806466 1846 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
I20260620 19:14:59.806471 1987 ts_tablet_manager.cc:1428] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:14:59.806931 1988 raft_consensus.cc:348] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.807075 1988 raft_consensus.cc:374] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:14:59.807116 1988 raft_consensus.cc:729] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1341f9f81264423484314fe2c8236bca, State: Initialized, Role: FOLLOWER
I20260620 19:14:59.807224 1988 consensus_queue.cc:260] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.807466 1988 ts_tablet_manager.cc:1428] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:14:59.807475 1978 heartbeater.cc:503] Master 127.19.66.254:36181 was elected leader, sending a full tablet report...
W20260620 19:14:59.891857 1847 tablet.cc:2369] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:14:59.899654 1991 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:14:59.899762 1991 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.900092 1991 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863)
I20260620 19:14:59.903015 1848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.900276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:38647 (local address 127.19.66.194:50543)
0620 19:14:59.900328 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.900339 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.900381 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.900532 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.900533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.900535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.900601 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.900603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.901329 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.901330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.902154 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.902157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.902763 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.902844 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.902848 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.902902 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.902943 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:14:59.903012 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.900339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:50543 (local address 127.19.66.195:38647)
0620 19:14:59.900387 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.900389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.900401 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:14:59.900435 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.900436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.900446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.900509 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.900665 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.901257 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902250 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.902692 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902746 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.902754 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.902917 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.902943 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:14:59.903380 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.900378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.19.66.194:56177)
0620 19:14:59.900539 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.900549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:14:59.900603 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:14:59.900753 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:14:59.900754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:14:59.900756 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:14:59.900822 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.900824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.901545 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.901546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.902348 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:14:59.902351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.902990 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:14:59.903058 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.903062 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:14:59.903265 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 19:14:59.903305 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260620 19:14:59.903379 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.900468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:56177 (local address 127.19.66.193:46863)
0620 19:14:59.900513 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.900515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.900606 (+ 91us) server_negotiation.cc:407] Connection header received
0620 19:14:59.900660 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.900660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.900688 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.900737 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:14:59.900912 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.901463 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902437 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.902921 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.902982 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.903066 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.903274 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.903317 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:14:59.903369 1932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446752e22e2b4d519843c4dfc7f7785f" candidate_uuid: "6b56617c010241fc80faac600e13dbed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1341f9f81264423484314fe2c8236bca" is_pre_election: true
I20260620 19:14:59.903534 1932 raft_consensus.cc:2454] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b56617c010241fc80faac600e13dbed in term 0.
I20260620 19:14:59.903790 1734 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed; no voters:
I20260620 19:14:59.903860 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446752e22e2b4d519843c4dfc7f7785f" candidate_uuid: "6b56617c010241fc80faac600e13dbed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f59b25a8d4e4276816672f463838be1" is_pre_election: true
I20260620 19:14:59.903920 1991 raft_consensus.cc:2781] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:14:59.903995 1991 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:14:59.904023 1991 raft_consensus.cc:3037] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.904016 1662 raft_consensus.cc:2454] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b56617c010241fc80faac600e13dbed in term 0.
I20260620 19:14:59.904722 1991 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.904832 1991 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [CANDIDATE]: Term 1 election: Requested vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863)
I20260620 19:14:59.904994 1932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446752e22e2b4d519843c4dfc7f7785f" candidate_uuid: "6b56617c010241fc80faac600e13dbed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1341f9f81264423484314fe2c8236bca"
I20260620 19:14:59.905016 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "446752e22e2b4d519843c4dfc7f7785f" candidate_uuid: "6b56617c010241fc80faac600e13dbed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f59b25a8d4e4276816672f463838be1"
I20260620 19:14:59.905068 1932 raft_consensus.cc:3037] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.905085 1662 raft_consensus.cc:3037] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:14:59.905721 1662 raft_consensus.cc:2454] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b56617c010241fc80faac600e13dbed in term 1.
I20260620 19:14:59.905792 1932 raft_consensus.cc:2454] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b56617c010241fc80faac600e13dbed in term 1.
I20260620 19:14:59.905869 1734 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1, 6b56617c010241fc80faac600e13dbed; no voters:
I20260620 19:14:59.905969 1991 raft_consensus.cc:2781] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:14:59.906106 1991 raft_consensus.cc:686] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 1 LEADER]: Becoming Leader. State: Replica: 6b56617c010241fc80faac600e13dbed, State: Running, Role: LEADER
I20260620 19:14:59.906186 1991 consensus_queue.cc:237] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:14:59.906970 1520 catalog_manager.cc:5495] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b56617c010241fc80faac600e13dbed (127.19.66.194). New cstate: current_term: 1 leader_uuid: "6b56617c010241fc80faac600e13dbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } health_report { overall_health: UNKNOWN } } }
I20260620 19:14:59.940382 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.938417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.19.66.194:38955)
0620 19:14:59.938475 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:14:59.938477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:14:59.938485 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:14:59.938519 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:14:59.938519 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:14:59.938536 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:14:59.938600 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:14:59.938685 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.939141 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.939816 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:14:59.940003 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:14:59.940051 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:14:59.940132 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:14:59.940228 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:14:59.940282 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:14:59.940295 (+ 13us) server_negotiation.cc:299] Negotiation successful
0620 19:14:59.940309 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:14:59.942004 1932 raft_consensus.cc:1261] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Refusing update from remote peer 6b56617c010241fc80faac600e13dbed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:59.942037 1662 raft_consensus.cc:1261] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Refusing update from remote peer 6b56617c010241fc80faac600e13dbed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:14:59.942340 1994 consensus_queue.cc:1035] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:59.942469 1991 consensus_queue.cc:1035] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:14:59.944559 1998 mvcc.cc:205] Tried to move back new op lower bound from 7299001958160052224 to 7299001958016315392. Current Snapshot: MvccSnapshot[applied={T|T < 7299001958160052224}]
I20260620 19:14:59.945488 1999 mvcc.cc:205] Tried to move back new op lower bound from 7299001958160052224 to 7299001958016315392. Current Snapshot: MvccSnapshot[applied={T|T < 7299001958160052224}]
I20260620 19:14:59.946652 1520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:59.946731 1520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:14:59.947326 1520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Sending DeleteTablet for 3 replicas of tablet 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:14:59.947597 1780 tablet_service.cc:1514] Processing DeleteTablet for tablet 446752e22e2b4d519843c4dfc7f7785f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:41790
I20260620 19:14:59.947639 1912 tablet_service.cc:1514] Processing DeleteTablet for tablet 446752e22e2b4d519843c4dfc7f7785f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:44914
I20260620 19:14:59.947696 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 446752e22e2b4d519843c4dfc7f7785f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:14:59 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:35494
I20260620 19:14:59.947813 2007 tablet_replica.cc:330] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: stopping tablet replica
I20260620 19:14:59.947803 1717 tablet_replica.cc:330] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: stopping tablet replica
I20260620 19:14:59.947866 1717 raft_consensus.cc:2229] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:59.947897 2007 raft_consensus.cc:2229] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:14:59.947928 1717 raft_consensus.cc:2258] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.947993 2007 raft_consensus.cc:2258] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.948144 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1558
I20260620 19:14:59.948263 1717 ts_tablet_manager.cc:1905] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:59.949244 2006 tablet_replica.cc:330] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: stopping tablet replica
I20260620 19:14:59.949340 2006 raft_consensus.cc:2229] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:14:59.949429 2006 raft_consensus.cc:2258] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:14:59.949847 2006 ts_tablet_manager.cc:1905] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:14:59.951437 2006 ts_tablet_manager.cc:1918] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:59.951506 2006 log.cc:1196] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/446752e22e2b4d519843c4dfc7f7785f
I20260620 19:14:59.951768 2006 ts_tablet_manager.cc:1939] T 446752e22e2b4d519843c4dfc7f7785f P 6b56617c010241fc80faac600e13dbed: Deleting consensus metadata
I20260620 19:14:59.952111 1506 catalog_manager.cc:4841] TS 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): tablet 446752e22e2b4d519843c4dfc7f7785f (table test-table [id=3cc31b71a92143d997bbdb1f1bf027f9]) successfully deleted
I20260620 19:14:59.953539 2007 ts_tablet_manager.cc:1905] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 19:14:59.953951 1505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:46863 (error 108)
I20260620 19:14:59.954066 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1714
I20260620 19:14:59.954820 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.954693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46863 (local address 127.0.0.1:35506)
0620 19:14:59.954743 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.954791 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:14:59.954942 1505 catalog_manager.cc:4584] TS 3f59b25a8d4e4276816672f463838be1 (127.19.66.193:46863): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 446752e22e2b4d519843c4dfc7f7785f: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46863: connect: Connection refused (error 111)
I20260620 19:14:59.959206 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1849
I20260620 19:14:59.959316 2007 ts_tablet_manager.cc:1918] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:14:59.959378 2007 log.cc:1196] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/446752e22e2b4d519843c4dfc7f7785f
I20260620 19:14:59.959615 2007 ts_tablet_manager.cc:1939] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Deleting consensus metadata
I20260620 19:14:59.963460 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 1487
I20260620 19:14:59.963578 1715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:14:59.963472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:38647 (local address 127.0.0.1:44918)
0620 19:14:59.963521 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:14:59.963546 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.195:38647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:14:59.967832 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--webserver_interface=127.19.66.254
--webserver_port=44553
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:00.046118 2009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:00.046312 2009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:00.046334 2009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:00.047459 2009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:15:00.047498 2009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:00.047511 2009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:00.047525 2009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:15:00.047549 2009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:15:00.049034 2009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=44553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:00.049258 2009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:00.049453 2009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:00.053910 2015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.053999 2017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.053963 2009 server_base.cc:1034] running on GCE node
W20260620 19:15:00.054160 2014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.054414 2009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:00.054661 2009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:00.055788 2009 hybrid_clock.cc:648] HybridClock initialized: now 1781982900055773 us; error 31 us; skew 500 ppm
Jun 20 19:15:00 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:00.058157 2009 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:15:00.058687 2009 webserver.cc:460] Webserver started at http://127.19.66.254:44553/ using document root <none> and password file <none>
I20260620 19:15:00.058907 2009 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:00.058950 2009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:00.060149 2009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:15:00.060699 2024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.060866 2009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.060947 2009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "66169377a3c1401fa3215d6f104a63d0"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "803694d7af537060e8a08d45f7c34522"
server_key_iv: "8b2f498ff216cac429b3348466529e22"
server_key_version: "encryptionkey@0"
I20260620 19:15:00.061271 2009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:00.073529 2009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:00.075691 2009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:00.075804 2009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:00.079536 2009 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:36181
I20260620 19:15:00.079571 2076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:36181 every 8 connection(s)
I20260620 19:15:00.079926 2009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:15:00.081573 2077 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:15:00.081837 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2009
I20260620 19:15:00.082073 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:46863
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=41413
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:00.082283 2077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap starting.
I20260620 19:15:00.090517 2077 log.cc:824] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:00.092674 2077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:00.092988 2077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0: Bootstrap complete.
I20260620 19:15:00.095023 2077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:15:00.095243 2077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Initialized, Role: FOLLOWER
I20260620 19:15:00.095392 2077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:15:00.095468 2077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:15:00.095515 2077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:15:00.095562 2077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:15:00.096520 2077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:15:00.096616 2077 leader_election.cc:302] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66169377a3c1401fa3215d6f104a63d0; no voters:
I20260620 19:15:00.096771 2077 leader_election.cc:288] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:15:00.096827 2081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:15:00.096973 2081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [term 3 LEADER]: Becoming Leader. State: Replica: 66169377a3c1401fa3215d6f104a63d0, State: Running, Role: LEADER
I20260620 19:15:00.097059 2077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:15:00.097059 2081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } }
I20260620 19:15:00.097378 2083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:15:00.097502 2083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:00.097386 2081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66169377a3c1401fa3215d6f104a63d0. Latest consensus state: current_term: 3 leader_uuid: "66169377a3c1401fa3215d6f104a63d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66169377a3c1401fa3215d6f104a63d0" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36181 } } }
I20260620 19:15:00.097712 2081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66169377a3c1401fa3215d6f104a63d0 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:00.097800 2088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:15:00.098685 2088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:15:00.098814 2088 catalog_manager.cc:1259] Loaded cluster ID: e010c0fa463f4d90ab4f67f28b3cabca
I20260620 19:15:00.098842 2088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 19:15:00.099109 2088 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcfd2a649ad google::LogMessage::Fail() at ??:0
@ 0x7fcfd2a6788a google::LogMessage::SendToLog() at ??:0
@ 0x7fcfd2a644de google::LogMessage::Flush() at ??:0
@ 0x7fcfd2a65a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcfda9890f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fcfda9928b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcfd49320d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcfd4927954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcfd578b6db start_thread at ??:0
@ 0x7fcfd156261f clone at ??:0
W20260620 19:15:00.172472 2079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:00.172667 2079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:00.172690 2079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:00.173846 2079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:00.173887 2079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:00.173913 2079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:15:00.175316 2079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:46863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=41413
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:00.175531 2079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:00.175746 2079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:00.180442 2113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.180434 2114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.180447 2116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.180697 2079 server_base.cc:1034] running on GCE node
I20260620 19:15:00.180922 2079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:00.181257 2079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:00.182428 2079 hybrid_clock.cc:648] HybridClock initialized: now 1781982900182409 us; error 33 us; skew 500 ppm
Jun 20 19:15:00 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:00.185420 2079 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:15:00.185946 2079 webserver.cc:460] Webserver started at http://127.19.66.193:41413/ using document root <none> and password file <none>
I20260620 19:15:00.186228 2079 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:00.186290 2079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:00.187937 2079 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:15:00.188737 2123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.188932 2079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:00.189038 2079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "3f59b25a8d4e4276816672f463838be1"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "e8c629f92cb3007dd060ec86830ed254"
server_key_iv: "6bb6969f69beac675135334f8e4f0c9d"
server_key_version: "encryptionkey@0"
I20260620 19:15:00.189498 2079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:00.211905 2079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:00.214382 2079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:00.214530 2079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:00.214783 2079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:00.215281 2130 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:15:00.216782 2079 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:15:00.216836 2079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:15:00.216904 2079 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:15:00.217542 2079 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:15:00.217578 2079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:15:00.217693 2130 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Bootstrap starting.
I20260620 19:15:00.219242 2130 log.cc:824] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:00.222090 2130 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:00.222373 2130 tablet_bootstrap.cc:492] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Bootstrap complete.
I20260620 19:15:00.222520 2130 ts_tablet_manager.cc:1397] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 19:15:00.224175 2130 raft_consensus.cc:348] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:00.224496 2130 raft_consensus.cc:729] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f59b25a8d4e4276816672f463838be1, State: Initialized, Role: FOLLOWER
I20260620 19:15:00.224673 2130 consensus_queue.cc:260] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:00.224968 2130 ts_tablet_manager.cc:1428] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260620 19:15:00.225162 2079 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:46863
I20260620 19:15:00.225226 2239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:46863 every 8 connection(s)
I20260620 19:15:00.225533 2079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:15:00.227257 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2079
I20260620 19:15:00.227481 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:38955
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=43491
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:00.307305 2243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:00.307502 2243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:00.307531 2243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:00.308650 2243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:00.308687 2243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:00.308722 2243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:15:00.310149 2243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:38955
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=43491
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:00.310385 2243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:00.310590 2243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:00.315163 2251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.315207 2249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.315410 2248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.315412 2243 server_base.cc:1034] running on GCE node
I20260620 19:15:00.315626 2243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:00.315845 2243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:00.316987 2243 hybrid_clock.cc:648] HybridClock initialized: now 1781982900316971 us; error 31 us; skew 500 ppm
Jun 20 19:15:00 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:00.319355 2243 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:15:00.319866 2243 webserver.cc:460] Webserver started at http://127.19.66.194:43491/ using document root <none> and password file <none>
I20260620 19:15:00.320089 2243 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:00.320147 2243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:00.321807 2243 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:00.322412 2258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.322651 2243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:00.322751 2243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "6b56617c010241fc80faac600e13dbed"
format_stamp: "Formatted at 2026-06-20 19:14:58 on dist-test-slave-nq8r"
server_key: "4f57f7fc339b42263e02ba9fb2744c07"
server_key_iv: "a1c95fc51ff3b81c3b4c696fd7c0a8ea"
server_key_version: "encryptionkey@0"
I20260620 19:15:00.323118 2243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260620 19:15:00.323603 2242 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:15:00.226174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:32789)
0620 19:15:00.226335 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.226366 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:00.226506 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:00.323410 (+ 96904us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260620 19:15:00.323798 2240 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:36181 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:36181: Client connection negotiation failed: client connection to 127.19.66.254:36181: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:15:00.324057 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.323959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:60549)
0620 19:15:00.323995 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.324032 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 19:15:00.324311 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.324235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:37379)
0620 19:15:00.324269 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.324290 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:15:00.324400 2240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:15:00.337481 2243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:00.339155 2243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:00.339278 2243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:00.339468 2243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:00.339743 2243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:00.339780 2243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.339807 2243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:00.339829 2243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.345335 2243 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:38955
I20260620 19:15:00.345389 2371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:38955 every 8 connection(s)
I20260620 19:15:00.345682 2243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:15:00.346596 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.346270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:33071)
0620 19:15:00.346397 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.346465 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
W20260620 19:15:00.346946 2372 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:36181 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:36181: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
I20260620 19:15:00.347169 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.347094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:44695)
0620 19:15:00.347129 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.347152 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 19:15:00.347360 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.347305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:35837)
0620 19:15:00.347333 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.347349 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 19:15:00.347435 2372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:15:00.352260 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2243
I20260620 19:15:00.352492 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:38647
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:36181
--webserver_port=44015
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:00.434075 2375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:00.434271 2375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:00.434294 2375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:00.435424 2375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:00.435463 2375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:00.435488 2375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:15:00.436982 2375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:38647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=44015
--tserver_master_addrs=127.19.66.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:00.437211 2375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:00.437412 2375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:00.442144 2381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:00.442154 2380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.442227 2375 server_base.cc:1034] running on GCE node
W20260620 19:15:00.442159 2383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:00.442549 2375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:00.442778 2375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:00.443917 2375 hybrid_clock.cc:648] HybridClock initialized: now 1781982900443908 us; error 27 us; skew 500 ppm
Jun 20 19:15:00 dist-test-slave-nq8r krb5kdc[942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:00.446523 2375 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:15:00.446940 2375 webserver.cc:460] Webserver started at http://127.19.66.195:44015/ using document root <none> and password file <none>
I20260620 19:15:00.447157 2375 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:00.447201 2375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:00.448441 2375 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:00.449031 2390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.449175 2375 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.449266 2375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "1341f9f81264423484314fe2c8236bca"
format_stamp: "Formatted at 2026-06-20 19:14:59 on dist-test-slave-nq8r"
server_key: "44f3477fd79da707dcf117ddb001122b"
server_key_iv: "07555beb0c586e7d252ef13b32b4569f"
server_key_version: "encryptionkey@0"
I20260620 19:15:00.449591 2375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:00.476121 2375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:00.478119 2375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:00.478233 2375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:00.478458 2375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:00.478890 2397 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 19:15:00.480252 2397 ts_tablet_manager.cc:1852] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 19:15:00.480525 2397 ts_tablet_manager.cc:1905] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:00.482414 2397 ts_tablet_manager.cc:1918] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:00.482537 2397 ts_tablet_manager.cc:1939] T 446752e22e2b4d519843c4dfc7f7785f P 1341f9f81264423484314fe2c8236bca: Deleting consensus metadata
I20260620 19:15:00.482725 2375 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 19:15:00.482771 2375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260620 19:15:00.482796 2375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:00.482820 2375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:00.488369 2375 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:38647
I20260620 19:15:00.488440 2504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:38647 every 8 connection(s)
I20260620 19:15:00.488705 2375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:15:00.489681 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.489298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:48373)
0620 19:15:00.489466 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.489541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
W20260620 19:15:00.489879 2505 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:36181 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:36181: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
I20260620 19:15:00.490141 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.490036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:47339)
0620 19:15:00.490072 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.490123 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 19:15:00.490379 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:00.490316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:36797)
0620 19:15:00.490347 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:00.490366 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:15:00.490458 2505 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:15:00.497390 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2375
I20260620 19:15:01.324954 2509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.324701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:46097)
0620 19:15:01.324830 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:01.324892 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260620 19:15:01.347949 2510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.347754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:36805)
0620 19:15:01.347858 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:01.347897 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260620 19:15:01.490965 2511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.490766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:39429)
0620 19:15:01.490881 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:01.490916 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260620 19:15:01.907356 2512 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:01.907467 2512 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:01.907819 2512 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
Jun 20 19:15:01 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.194@KRBTEST.COM
Jun 20 19:15:01 dist-test-slave-nq8r krb5kdc[942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982900, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.195@KRBTEST.COM
I20260620 19:15:01.919625 2515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.908093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:58321 (local address 127.19.66.195:38647)
0620 19:15:01.908254 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:01.908268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:01.908291 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:15:01.908939 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:01.908942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:01.908983 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:01.909094 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:01.910226 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.911504 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913024 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.913675 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913787 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:01.916634 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:01.916638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:01.916638 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:01.916648 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:01.918135 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918495 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:01.918499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:01.918499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:01.918583 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918777 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:01.918778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:01.918778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:01.919041 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:01.919140 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:01.919435 (+ 295us) server_negotiation.cc:299] Negotiation successful
0620 19:15:01.919536 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260620 19:15:01.919730 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.907981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:38647 (local address 127.19.66.193:58321)
0620 19:15:01.908183 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:01.908190 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:01.908213 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:01.909154 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:01.909157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:01.909625 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:01.910129 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:01.910131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.911612 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.911613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:01.912945 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:01.912948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.913768 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:01.913933 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:01.914511 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:01.914522 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:01.916526 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:01.918211 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:01.918212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:01.918427 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:01.918640 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:01.918642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:01.918724 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:01.919125 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:01.919132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:01.919289 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:15:01.919399 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:15:01.919588 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":108,"thread_start_us":44,"threads_started":1}
I20260620 19:15:01.919795 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.907981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:38955 (local address 127.19.66.193:53215)
0620 19:15:01.908084 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:01.908092 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:01.908136 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:01.909154 (+ 1018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:01.909157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:01.909623 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:01.909960 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:01.909970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.911626 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.911627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:01.912945 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:01.912948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.913765 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:01.913933 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:01.914561 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:01.914565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:01.916252 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:01.918127 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:01.918135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:01.918390 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:01.918597 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:01.918598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:01.918704 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:01.919228 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:01.919229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:01.919289 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 19:15:01.919399 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 19:15:01.919650 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":165,"thread_start_us":48,"threads_started":1}
I20260620 19:15:01.920064 2516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:01.908122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:53215 (local address 127.19.66.194:38955)
0620 19:15:01.908254 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:01.908268 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:01.908291 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:15:01.908866 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:01.908870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:01.908935 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:01.909099 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:01.910081 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.911503 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913024 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:01.913674 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:01.913788 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:01.916370 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:01.916374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:01.916374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:01.916384 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:01.918051 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918459 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:01.918462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:01.918462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:01.918541 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:01.918834 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:01.918835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:01.918836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:01.919119 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:01.919202 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:01.919833 (+ 631us) server_negotiation.cc:299] Negotiation successful
0620 19:15:01.919966 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":43,"threads_started":1}
W20260620 19:15:01.920722 2126 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
W20260620 19:15:01.920821 2125 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:15:01.920905 2125 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1; no voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed
I20260620 19:15:01.921017 2512 raft_consensus.cc:2726] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:15:02.325536 2514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:02.325415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:33019)
0620 19:15:02.325464 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:02.325496 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:15:02.348547 2517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:02.348354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:35533)
0620 19:15:02.348461 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:02.348504 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260620 19:15:02.491637 2518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:02.491420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:48967)
0620 19:15:02.491560 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:02.491599 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":58,"threads_started":1}
I20260620 19:15:03.326126 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:03.325920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:60335)
0620 19:15:03.326040 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:03.326079 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260620 19:15:03.349159 2520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:03.348970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:42783)
0620 19:15:03.349080 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:03.349115 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260620 19:15:03.492297 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:03.492075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:60117)
0620 19:15:03.492214 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:03.492253 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260620 19:15:03.986609 2522 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:03.986706 2522 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:03.986868 2522 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
W20260620 19:15:03.987250 2125 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
W20260620 19:15:03.987344 2126 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:15:03.987393 2126 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1; no voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed
I20260620 19:15:03.987454 2522 raft_consensus.cc:2726] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:15:04.326761 2523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:04.326513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:50105)
0620 19:15:04.326677 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:04.326715 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":57,"threads_started":1}
I20260620 19:15:04.349764 2524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:04.349545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:34915)
0620 19:15:04.349666 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:04.349705 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260620 19:15:04.493054 2525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:04.492792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:41497)
0620 19:15:04.492961 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:04.493008 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260620 19:15:05.327459 2526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:05.327253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:32777)
0620 19:15:05.327372 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:05.327414 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260620 19:15:05.350391 2527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:05.350205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:52801)
0620 19:15:05.350312 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:05.350351 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260620 19:15:05.493752 2528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:05.493522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:33223)
0620 19:15:05.493663 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:05.493707 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260620 19:15:06.328063 2529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:06.327863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:43075)
0620 19:15:06.327977 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:06.328014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
I20260620 19:15:06.350932 2530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:06.350751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:43473)
0620 19:15:06.350859 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:06.350890 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260620 19:15:06.494396 2531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:06.494195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:56291)
0620 19:15:06.494315 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:06.494353 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 19:15:07.260421 2532 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:07.260524 2532 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:07.260668 2532 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
W20260620 19:15:07.261065 2125 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
W20260620 19:15:07.261137 2126 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:15:07.261197 2126 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1; no voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed
I20260620 19:15:07.261272 2532 raft_consensus.cc:2726] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:15:07.328718 2533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:07.328493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:52607)
0620 19:15:07.328623 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:07.328667 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 19:15:07.351516 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:07.351326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:50669)
0620 19:15:07.351433 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:07.351461 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260620 19:15:07.495076 2535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:07.494868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:44515)
0620 19:15:07.494988 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:07.495031 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260620 19:15:08.329432 2536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:08.329200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:53337)
0620 19:15:08.329326 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:08.329367 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260620 19:15:08.352195 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:08.351943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:33715)
0620 19:15:08.352104 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:08.352150 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260620 19:15:08.495865 2538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:08.495627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:48861)
0620 19:15:08.495772 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:08.495821 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260620 19:15:09.330091 2539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:09.329878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:33565)
0620 19:15:09.330008 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:09.330047 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260620 19:15:09.352775 2540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:09.352573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:48281)
0620 19:15:09.352693 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:09.352730 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260620 19:15:09.496556 2541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:09.496327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:54833)
0620 19:15:09.496467 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:09.496511 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":60,"threads_started":1}
I20260620 19:15:10.330789 2542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:10.330545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:48205)
0620 19:15:10.330695 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:10.330741 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":74,"threads_started":1}
I20260620 19:15:10.353389 2543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:10.353171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:43711)
0620 19:15:10.353312 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:10.353346 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260620 19:15:10.497272 2544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:10.497051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:37779)
0620 19:15:10.497171 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:10.497216 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 19:15:11.100016 2545 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:11.100117 2545 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:11.100256 2545 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
W20260620 19:15:11.100639 2125 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
W20260620 19:15:11.100719 2126 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:15:11.100749 2126 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1; no voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed
I20260620 19:15:11.100832 2545 raft_consensus.cc:2726] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:15:11.331420 2546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:11.331222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:34899)
0620 19:15:11.331335 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:11.331373 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260620 19:15:11.353984 2547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:11.353790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:43327)
0620 19:15:11.353910 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:11.353942 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":56,"threads_started":1}
I20260620 19:15:11.497932 2548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:11.497719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:37303)
0620 19:15:11.497851 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:11.497890 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260620 19:15:12.332120 2549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:12.331869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:37035)
0620 19:15:12.332030 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:12.332072 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260620 19:15:12.354635 2550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:12.354414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:32841)
0620 19:15:12.354547 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:12.354588 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260620 19:15:12.498615 2551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:12.498388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:51519)
0620 19:15:12.498527 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:12.498569 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260620 19:15:13.332890 2552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:13.332614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:44759)
0620 19:15:13.332765 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:13.332809 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260620 19:15:13.355222 2553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:13.355013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:38011)
0620 19:15:13.355146 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:13.355182 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":84,"threads_started":1}
I20260620 19:15:13.499311 2554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:13.499065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:34023)
0620 19:15:13.499221 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:13.499262 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260620 19:15:14.013847 2555 raft_consensus.cc:482] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:14.013938 2555 raft_consensus.cc:504] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1341f9f81264423484314fe2c8236bca" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 38647 } } peers { permanent_uuid: "6b56617c010241fc80faac600e13dbed" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 38955 } } peers { permanent_uuid: "3f59b25a8d4e4276816672f463838be1" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46863 } }
I20260620 19:15:14.014096 2555 leader_election.cc:288] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647), 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955)
W20260620 19:15:14.014408 2126 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b56617c010241fc80faac600e13dbed (127.19.66.194:38955): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
W20260620 19:15:14.014485 2125 leader_election.cc:341] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1341f9f81264423484314fe2c8236bca (127.19.66.195:38647): Not found: Tablet not found: 446752e22e2b4d519843c4dfc7f7785f
I20260620 19:15:14.014540 2125 leader_election.cc:302] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f59b25a8d4e4276816672f463838be1; no voters: 1341f9f81264423484314fe2c8236bca, 6b56617c010241fc80faac600e13dbed
I20260620 19:15:14.014616 2555 raft_consensus.cc:2726] T 446752e22e2b4d519843c4dfc7f7785f P 3f59b25a8d4e4276816672f463838be1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:15:14.333570 2556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:14.333373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:38659)
0620 19:15:14.333489 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:14.333523 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260620 19:15:14.355803 2557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:14.355592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:37193)
0620 19:15:14.355719 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:14.355759 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260620 19:15:14.499945 2558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:14.499721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:34357)
0620 19:15:14.499865 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:14.499900 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260620 19:15:15.334316 2559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:15.334097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.193:37045)
0620 19:15:15.334219 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:15.334252 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260620 19:15:15.356419 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:15.356225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.194:49841)
0620 19:15:15.356342 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:15.356375 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260620 19:15:15.500705 2561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:15.500467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36181 (local address 127.19.66.195:51257)
0620 19:15:15.500614 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:15.500658 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:36181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":64,"threads_started":1}
I20260620 19:15:15.501178 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2079
I20260620 19:15:15.505297 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2243
I20260620 19:15:15.508805 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2375
2026-06-20T19:15:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18028 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:15:15 dist-test-slave-nq8r krb5kdc[2564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:15:15 dist-test-slave-nq8r krb5kdc[2564](info): set up 2 sockets
Jun 20 19:15:15 dist-test-slave-nq8r krb5kdc[2564](info): commencing operation
krb5kdc: starting...
W20260620 19:15:16.452564 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:15:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:15:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:15:16.559510 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:16.638952 2577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:16.639163 2577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:16.639183 2577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:16.640357 2577 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:15:16.640398 2577 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:15:16.640410 2577 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:15:16.640420 2577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:15:16.640431 2577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:16.640444 2577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:16.640460 2577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:15:16.640483 2577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:15:16.642011 2577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:16.642251 2577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:16.642457 2577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:16.646952 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.646993 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.646970 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:16.647277 2577 server_base.cc:1034] running on GCE node
I20260620 19:15:16.647511 2577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:16.647840 2577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:16.648989 2577 hybrid_clock.cc:648] HybridClock initialized: now 1781982916648973 us; error 23 us; skew 500 ppm
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:16.651247 2577 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:15:16.651684 2577 webserver.cc:460] Webserver started at http://127.19.66.254:35257/ using document root <none> and password file <none>
I20260620 19:15:16.651898 2577 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:16.651945 2577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:16.652069 2577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:15:16.653144 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "083ca6d1218749aca9a64c1133bfc4c3"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "dc3d17243c284a7bdf06239bb0949939"
server_key_iv: "03432133c8e0cc0f9d9b0c125c0bca55"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.653484 2577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "083ca6d1218749aca9a64c1133bfc4c3"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "dc3d17243c284a7bdf06239bb0949939"
server_key_iv: "03432133c8e0cc0f9d9b0c125c0bca55"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.654676 2577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:15:16.655324 2592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.655468 2577 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:16.655547 2577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "083ca6d1218749aca9a64c1133bfc4c3"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "dc3d17243c284a7bdf06239bb0949939"
server_key_iv: "03432133c8e0cc0f9d9b0c125c0bca55"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.655625 2577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:16.664754 2577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:16.666483 2577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:16.666616 2577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:16.670228 2577 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:40905
I20260620 19:15:16.670255 2644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:40905 every 8 connection(s)
I20260620 19:15:16.670567 2577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:15:16.671360 2645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:16.673574 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap starting.
I20260620 19:15:16.673810 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2577
I20260620 19:15:16.673888 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:15:16.674074 19723 external_mini_cluster.cc:1428] Setting key f6173d0e16026051f52c09b19abeb313
I20260620 19:15:16.674144 2645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:16.674414 2645 log.cc:824] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:16.675107 2645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: No bootstrap required, opened a new log
I20260620 19:15:16.677206 2645 raft_consensus.cc:348] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:16.677388 2645 raft_consensus.cc:374] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:16.677430 2645 raft_consensus.cc:729] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Initialized, Role: FOLLOWER
I20260620 19:15:16.677553 2645 consensus_queue.cc:260] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:16.677624 2645 raft_consensus.cc:388] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:15:16.677657 2645 raft_consensus.cc:482] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:15:16.677698 2645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:16.678292 2645 raft_consensus.cc:504] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:16.678386 2645 leader_election.cc:302] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 083ca6d1218749aca9a64c1133bfc4c3; no voters:
I20260620 19:15:16.678527 2645 leader_election.cc:288] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:15:16.678565 2650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:15:16.678735 2650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 LEADER]: Becoming Leader. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Running, Role: LEADER
I20260620 19:15:16.678877 2645 sys_catalog.cc:564] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:15:16.678898 2650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:16.679227 2650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 083ca6d1218749aca9a64c1133bfc4c3. Latest consensus state: current_term: 1 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:16.679456 2650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:16.679390 2651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:16.679864 2651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:16.680016 2659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:16.680485 2659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:15:16.682713 2659 catalog_manager.cc:1347] Generated new cluster ID: 8df66e70df9f420591be8cae72a89bc7
I20260620 19:15:16.682763 2659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:15:16.684283 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:16.674518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.19.66.254:40905)
0620 19:15:16.674678 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:16.674688 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:16.674720 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:15:16.675160 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:16.675166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:16.675226 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:16.675357 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:16.676089 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.677014 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.678028 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.678500 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.678592 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:16.681187 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:16.681198 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:16.681199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:16.681212 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:16.682959 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:16.683252 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:16.683255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:16.683255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:16.683357 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:16.683523 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:16.683524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:16.683524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:16.683747 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:16.683893 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:16.683984 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:15:16.684110 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
I20260620 19:15:16.701995 2659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:15:16.702548 2659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:15:16.711185 2659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Generated new TSK 0
I20260620 19:15:16.711400 2659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:15:16.737197 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:16.815026 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:16.815214 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:16.815237 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:16.816375 2671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:16.816414 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:16.816439 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:15:16.817874 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:16.818109 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:16.818305 2671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:16.822686 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.822781 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.822916 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:16.822983 2671 server_base.cc:1034] running on GCE node
I20260620 19:15:16.823170 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:16.823398 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:16.824545 2671 hybrid_clock.cc:648] HybridClock initialized: now 1781982916824531 us; error 27 us; skew 500 ppm
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:16.826881 2671 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:15:16.827343 2671 webserver.cc:460] Webserver started at http://127.19.66.193:44551/ using document root <none> and password file <none>
I20260620 19:15:16.827538 2671 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:16.827582 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:16.827705 2671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:15:16.828708 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "286bf1a9b21b492784c4fcbdbed29002"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "96ab29af1744f4c253beb3ffeb138365"
server_key_iv: "f63cd563b54e9fa791b91c510ae1c794"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.829043 2671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "286bf1a9b21b492784c4fcbdbed29002"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "96ab29af1744f4c253beb3ffeb138365"
server_key_iv: "f63cd563b54e9fa791b91c510ae1c794"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.830206 2671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:15:16.830809 2686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.830961 2671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:15:16.831043 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "286bf1a9b21b492784c4fcbdbed29002"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "96ab29af1744f4c253beb3ffeb138365"
server_key_iv: "f63cd563b54e9fa791b91c510ae1c794"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.831106 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:16.849843 2671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:16.851847 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:16.851971 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:16.852138 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:16.852424 2671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:16.852456 2671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.852478 2671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:16.852500 2671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.857683 2671 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:37803
I20260620 19:15:16.857729 2799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:37803 every 8 connection(s)
I20260620 19:15:16.858006 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:15:16.861285 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2671
I20260620 19:15:16.861372 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:15:16.861545 19723 external_mini_cluster.cc:1428] Setting key bc8103853d6edee8799499d5c139a94f
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:16.866618 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:16.858676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:53105 (local address 127.19.66.254:40905)
0620 19:15:16.858716 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:16.858718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:16.858940 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:15:16.859439 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:16.859441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:16.859452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:16.859499 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:16.860426 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.860965 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.862010 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.862205 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.862259 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:16.864160 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:16.864161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:16.864162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:16.864169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:16.865413 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:16.865764 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:16.865765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:16.865766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:16.865820 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:16.866033 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:16.866034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:16.866034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:16.866141 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:16.866196 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:16.866440 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 19:15:16.866530 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:15:16.866798 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:16.858661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:53105)
0620 19:15:16.858833 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:16.858866 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:16.859348 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:16.859581 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:16.859585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:16.860143 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:16.860338 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:16.860343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.861040 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.861041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:16.861934 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:16.861937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.862271 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.862272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:16.862402 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:16.862849 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:16.862858 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:16.864079 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:16.865487 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:16.865494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:16.865496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:16.865704 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:16.865873 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:16.865873 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:16.865874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:16.865981 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:16.866194 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:16.866201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:16.866328 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 19:15:16.866412 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 19:15:16.866557 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260620 19:15:16.867250 2800 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:16.867336 2800 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:16.867497 2800 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:16.867998 2609 ts_manager.cc:194] Registered new tserver with Master: 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803)
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
I20260620 19:15:16.868634 2609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:53105
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:15:16.881233 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:16.959048 2805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:16.959246 2805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:16.959267 2805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:16.960434 2805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:16.960476 2805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:16.960505 2805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:15:16.961910 2805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:16.962112 2805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:16.962314 2805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:16.966744 2810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.966809 2811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:16.966797 2813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:16.967008 2805 server_base.cc:1034] running on GCE node
I20260620 19:15:16.967231 2805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:16.967486 2805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:16.968626 2805 hybrid_clock.cc:648] HybridClock initialized: now 1781982916968615 us; error 23 us; skew 500 ppm
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:16.970883 2805 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:15:16.971307 2805 webserver.cc:460] Webserver started at http://127.19.66.194:38189/ using document root <none> and password file <none>
I20260620 19:15:16.971498 2805 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:16.971542 2805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:16.971662 2805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:15:16.972625 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "881c355127524514b175bb4a19aa8693"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "da84918022edc6df2ca3372154fff155"
server_key_iv: "070ff827225f2c1bd6aaa5244576adde"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.972966 2805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "881c355127524514b175bb4a19aa8693"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "da84918022edc6df2ca3372154fff155"
server_key_iv: "070ff827225f2c1bd6aaa5244576adde"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.974248 2805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:15:16.974972 2820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.975121 2805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:15:16.975207 2805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "881c355127524514b175bb4a19aa8693"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "da84918022edc6df2ca3372154fff155"
server_key_iv: "070ff827225f2c1bd6aaa5244576adde"
server_key_version: "encryptionkey@0"
I20260620 19:15:16.975272 2805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:16.987673 2805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:16.989655 2805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:16.989800 2805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:16.990006 2805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:16.990321 2805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:16.990356 2805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.990384 2805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:16.990408 2805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:16.995787 2805 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:41829
I20260620 19:15:16.995836 2933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:41829 every 8 connection(s)
I20260620 19:15:16.996129 2805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:15:16 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982916, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:17.004536 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:16.996792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:36333 (local address 127.19.66.254:40905)
0620 19:15:16.996834 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:16.996836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:16.997050 (+ 214us) server_negotiation.cc:407] Connection header received
0620 19:15:16.997560 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:16.997561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:16.997573 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:16.997621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:16.998561 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.999053 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.000021 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.000208 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.000262 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.002257 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:17.002259 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:17.002259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:17.002267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:17.003328 (+ 1061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.003671 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.003672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.003673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.003725 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.003934 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.003935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.003935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.004045 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:17.004099 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.004362 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.004441 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:15:17.004719 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:16.996759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:36333)
0620 19:15:16.996941 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:16.996976 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:16.997466 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:16.997692 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:16.997696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:16.998278 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:16.998468 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:16.998473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:16.999135 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:16.999136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:16.999938 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:16.999940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.000281 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.000282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.000403 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.000878 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:17.000889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:17.002169 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:17.003398 (+ 1229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:17.003405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:17.003407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:17.003609 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:17.003777 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:17.003777 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:17.003777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:17.003880 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:17.004096 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:17.004102 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:17.004236 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.004332 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.004498 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":94,"threads_started":1}
I20260620 19:15:17.005257 2934 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:17.005350 2934 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:17.005355 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2805
I20260620 19:15:17.005452 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:15:17.005538 2934 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:17.005654 19723 external_mini_cluster.cc:1428] Setting key f0aebbaa08c7ecf506891d0b7ed5db7f
I20260620 19:15:17.005899 2609 ts_manager.cc:194] Registered new tserver with Master: 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
I20260620 19:15:17.006422 2609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:36333
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:15:17.026949 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:17.104817 2939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:17.105034 2939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:17.105055 2939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:17.106124 2939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:17.106159 2939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:17.106191 2939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:15:17.107555 2939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:17.107738 2939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:17.107903 2939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:17.112524 2947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:17.112519 2945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:17.112730 2939 server_base.cc:1034] running on GCE node
W20260620 19:15:17.112524 2944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:17.112995 2939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:17.113224 2939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:17.114382 2939 hybrid_clock.cc:648] HybridClock initialized: now 1781982917114366 us; error 28 us; skew 500 ppm
Jun 20 19:15:17 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:17.116715 2939 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:15:17.117136 2939 webserver.cc:460] Webserver started at http://127.19.66.195:40027/ using document root <none> and password file <none>
I20260620 19:15:17.117326 2939 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:17.117367 2939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:17.117449 2939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:15:17.118431 2939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "c59822ff37a34691b98a99ea1114ef68"
format_stamp: "Formatted at 2026-06-20 19:15:17 on dist-test-slave-nq8r"
server_key: "bf6c1f223fab8b3c30271ea1aef29487"
server_key_iv: "c4b230dc08974b7141dcb13b9d683e46"
server_key_version: "encryptionkey@0"
I20260620 19:15:17.118708 2939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "c59822ff37a34691b98a99ea1114ef68"
format_stamp: "Formatted at 2026-06-20 19:15:17 on dist-test-slave-nq8r"
server_key: "bf6c1f223fab8b3c30271ea1aef29487"
server_key_iv: "c4b230dc08974b7141dcb13b9d683e46"
server_key_version: "encryptionkey@0"
I20260620 19:15:17.119823 2939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:15:17.120436 2954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:17.120587 2939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:17.120667 2939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "c59822ff37a34691b98a99ea1114ef68"
format_stamp: "Formatted at 2026-06-20 19:15:17 on dist-test-slave-nq8r"
server_key: "bf6c1f223fab8b3c30271ea1aef29487"
server_key_iv: "c4b230dc08974b7141dcb13b9d683e46"
server_key_version: "encryptionkey@0"
I20260620 19:15:17.120726 2939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:17.143843 2939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:17.145830 2939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:17.145941 2939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:17.146118 2939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:17.146389 2939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:17.146420 2939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:17.146440 2939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:17.146456 2939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:17.151744 2939 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33227
I20260620 19:15:17.151814 3067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33227 every 8 connection(s)
I20260620 19:15:17.152084 2939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:15:17 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982917, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:17.160625 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.152801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:51923 (local address 127.19.66.254:40905)
0620 19:15:17.152888 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.152890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.153040 (+ 150us) server_negotiation.cc:407] Connection header received
0620 19:15:17.153543 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.153545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.153557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.153609 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:17.154519 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.155038 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.156004 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.156201 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.156255 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.158265 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:17.158267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:17.158267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:17.158275 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:17.159428 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.159774 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.159775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.159775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.159829 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.160039 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.160040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.160040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.160151 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:17.160206 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.160457 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.160536 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 19:15:17.160786 3070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.152728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:51923)
0620 19:15:17.152916 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.152961 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.153444 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.153680 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.153684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.154233 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:17.154424 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.154428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.155122 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.155123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.155918 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.155921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.156269 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.156270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.156399 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.156828 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:17.156836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:17.158161 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:17.159498 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:17.159505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:17.159507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:17.159710 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:17.159881 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:17.159882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:17.159882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:17.159985 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:17.160210 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:17.160216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:17.160345 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.160432 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.160579 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260620 19:15:17.161203 3068 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:17.161315 3068 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:17.161484 3068 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:17.161491 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 2939
I20260620 19:15:17.161569 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:15:17.161764 19723 external_mini_cluster.cc:1428] Setting key 954635081581a1161a0d348b84d8bead
I20260620 19:15:17.161826 2609 ts_manager.cc:194] Registered new tserver with Master: c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227)
I20260620 19:15:17.162223 2609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:51923
I20260620 19:15:17.162528 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:15:17.169150 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.163799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.19.66.254:40905)
0620 19:15:17.163837 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.163839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.163973 (+ 134us) server_negotiation.cc:407] Connection header received
0620 19:15:17.164025 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.164026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.164035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.164075 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:17.164579 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.165188 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.165884 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.166064 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.166115 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.167143 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:17.167144 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:17.167145 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:17.167151 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:17.168177 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.168466 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.168467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.168467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.168513 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:17.168664 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:17.168665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:17.168665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:17.168798 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:17.168904 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.168995 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.169068 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:15:17.170403 2608 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:15:17.172269 2608 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:15:17.177740 3078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33227 (local address 127.0.0.1:39090)
0620 19:15:17.174395 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.174437 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.174499 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.174696 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.174699 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.174701 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:17.174751 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.174752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.175762 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.175763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.176762 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.176765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.177436 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.177437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.177519 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.177533 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.177597 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.177650 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":52,"threads_started":1}
I20260620 19:15:17.177731 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.19.66.195:33227)
0620 19:15:17.174538 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.174542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.174551 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:15:17.174589 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.174591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.174608 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.174685 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:17.174815 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.175670 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.176842 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.177359 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.177424 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.177490 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.177616 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.177655 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260620 19:15:17.178414 3002 tablet_service.cc:1467] Processing CreateTablet for tablet feae23a0ac2a4822b5a4700512820c41 (DEFAULT_TABLE table=test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:17.178823 3002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feae23a0ac2a4822b5a4700512820c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:17.178841 3077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41829 (local address 127.0.0.1:55930)
0620 19:15:17.174482 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.174493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.174543 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.174958 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.174958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.174963 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:17.175009 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.175010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.176327 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.176328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.177384 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.177386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.178613 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.178615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.178699 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.178703 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.178737 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.178776 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260620 19:15:17.179044 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.19.66.194:41829)
0620 19:15:17.174807 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.174810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.174819 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:15:17.174851 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.174853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.174866 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.175074 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:17.175142 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.176221 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.177511 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.178521 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.178664 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.178730 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.178765 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.178804 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":74,"threads_started":1}
I20260620 19:15:17.179411 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:37803 (local address 127.0.0.1:37504)
0620 19:15:17.174611 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.174619 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.174653 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.174957 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.174958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.174960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:17.175009 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.175011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.177117 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.177118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.178174 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.178177 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.179186 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.179187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.179251 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.179255 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.179312 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.179351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":82,"threads_started":1}
I20260620 19:15:17.179451 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.174379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37504 (local address 127.19.66.193:37803)
0620 19:15:17.174797 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.174801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.174810 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:15:17.174846 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.174848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.174862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.175075 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:17.175144 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.176225 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.178256 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.179080 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.179147 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.179197 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.179334 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.179377 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":82,"threads_started":1}
I20260620 19:15:17.180094 2734 tablet_service.cc:1467] Processing CreateTablet for tablet feae23a0ac2a4822b5a4700512820c41 (DEFAULT_TABLE table=test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:17.180450 2734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feae23a0ac2a4822b5a4700512820c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:17.181423 2868 tablet_service.cc:1467] Processing CreateTablet for tablet feae23a0ac2a4822b5a4700512820c41 (DEFAULT_TABLE table=test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:17.181694 2868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet feae23a0ac2a4822b5a4700512820c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:17.182416 3082 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Bootstrap starting.
I20260620 19:15:17.183053 3082 tablet_bootstrap.cc:654] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:17.183589 3082 log.cc:824] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:17.183606 3083 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap starting.
I20260620 19:15:17.184295 3082 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: No bootstrap required, opened a new log
I20260620 19:15:17.184309 3083 tablet_bootstrap.cc:654] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:17.184361 3082 ts_tablet_manager.cc:1397] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:15:17.184836 3083 log.cc:824] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:17.184998 3085 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Bootstrap starting.
I20260620 19:15:17.185602 3085 tablet_bootstrap.cc:654] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:17.185747 3083 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: No bootstrap required, opened a new log
I20260620 19:15:17.185812 3083 ts_tablet_manager.cc:1397] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:15:17.186141 3085 log.cc:824] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:17.186167 3082 raft_consensus.cc:348] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.186345 3082 raft_consensus.cc:374] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:17.186389 3082 raft_consensus.cc:729] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59822ff37a34691b98a99ea1114ef68, State: Initialized, Role: FOLLOWER
I20260620 19:15:17.186523 3082 consensus_queue.cc:260] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.186771 3082 ts_tablet_manager.cc:1428] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:15:17.186790 3068 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
I20260620 19:15:17.186968 3085 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: No bootstrap required, opened a new log
I20260620 19:15:17.187022 3085 ts_tablet_manager.cc:1397] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:15:17.188176 3083 raft_consensus.cc:348] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.188302 3083 raft_consensus.cc:374] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:17.188323 3083 raft_consensus.cc:729] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bf1a9b21b492784c4fcbdbed29002, State: Initialized, Role: FOLLOWER
I20260620 19:15:17.188410 3083 consensus_queue.cc:260] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.188639 3083 ts_tablet_manager.cc:1428] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 19:15:17.188728 2800 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
I20260620 19:15:17.188560 3085 raft_consensus.cc:348] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.189041 3085 raft_consensus.cc:374] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:17.189085 3085 raft_consensus.cc:729] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881c355127524514b175bb4a19aa8693, State: Initialized, Role: FOLLOWER
I20260620 19:15:17.189186 3085 consensus_queue.cc:260] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.189419 3085 ts_tablet_manager.cc:1428] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:15:17.189452 2934 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
I20260620 19:15:17.238515 3090 raft_consensus.cc:482] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:17.238629 3090 raft_consensus.cc:504] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.238998 3090 leader_election.cc:288] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803)
I20260620 19:15:17.241736 3079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.239238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:44805 (local address 127.19.66.195:33227)
0620 19:15:17.239279 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.239281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.239291 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:15:17.239356 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.239357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.239366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.239431 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:17.239583 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.240089 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241041 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.241424 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241476 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.241483 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.241632 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.241670 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:15:17.241756 3081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.239238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:49455 (local address 127.19.66.193:37803)
0620 19:15:17.239296 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.239297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.239448 (+ 151us) server_negotiation.cc:407] Connection header received
0620 19:15:17.239481 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.239482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.239496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.239552 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:17.239689 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.240160 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241097 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.241469 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241539 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.241548 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.241664 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.241700 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:15:17.241753 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.239182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33227 (local address 127.19.66.194:44805)
0620 19:15:17.239237 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.239248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.239289 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.239440 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.239441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.239443 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:17.239519 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.239521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.240157 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.240158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.240965 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.240968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.241494 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.241569 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.241572 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.241614 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.241649 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":5}
I20260620 19:15:17.241770 3091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.239206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:37803 (local address 127.19.66.194:49455)
0620 19:15:17.239370 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.239381 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:17.239438 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:17.239559 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:17.239560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:17.239562 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:17.239631 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.239632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.240236 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.240236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.241020 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:17.241023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.241544 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.241545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:17.241598 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.241602 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:15:17.241652 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:15:17.241687 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260620 19:15:17.242105 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feae23a0ac2a4822b5a4700512820c41" candidate_uuid: "881c355127524514b175bb4a19aa8693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59822ff37a34691b98a99ea1114ef68" is_pre_election: true
I20260620 19:15:17.242234 2754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feae23a0ac2a4822b5a4700512820c41" candidate_uuid: "881c355127524514b175bb4a19aa8693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bf1a9b21b492784c4fcbdbed29002" is_pre_election: true
I20260620 19:15:17.242357 3022 raft_consensus.cc:2454] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881c355127524514b175bb4a19aa8693 in term 0.
I20260620 19:15:17.242444 2754 raft_consensus.cc:2454] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881c355127524514b175bb4a19aa8693 in term 0.
I20260620 19:15:17.242652 2824 leader_election.cc:302] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002, 881c355127524514b175bb4a19aa8693; no voters:
I20260620 19:15:17.242779 3090 raft_consensus.cc:2781] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:15:17.242851 3090 raft_consensus.cc:482] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:15:17.242869 3090 raft_consensus.cc:3037] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:17.243537 3090 raft_consensus.cc:504] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.243649 3090 leader_election.cc:288] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [CANDIDATE]: Term 1 election: Requested vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803)
I20260620 19:15:17.243794 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feae23a0ac2a4822b5a4700512820c41" candidate_uuid: "881c355127524514b175bb4a19aa8693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59822ff37a34691b98a99ea1114ef68"
I20260620 19:15:17.243804 2754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "feae23a0ac2a4822b5a4700512820c41" candidate_uuid: "881c355127524514b175bb4a19aa8693" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286bf1a9b21b492784c4fcbdbed29002"
I20260620 19:15:17.243870 3022 raft_consensus.cc:3037] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:17.243870 2754 raft_consensus.cc:3037] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:17.244695 2754 raft_consensus.cc:2454] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881c355127524514b175bb4a19aa8693 in term 1.
I20260620 19:15:17.244695 3022 raft_consensus.cc:2454] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881c355127524514b175bb4a19aa8693 in term 1.
I20260620 19:15:17.244890 2823 leader_election.cc:302] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68; no voters:
I20260620 19:15:17.244973 3090 raft_consensus.cc:2781] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:15:17.245131 3090 raft_consensus.cc:686] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 LEADER]: Becoming Leader. State: Replica: 881c355127524514b175bb4a19aa8693, State: Running, Role: LEADER
I20260620 19:15:17.245241 3090 consensus_queue.cc:237] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:17.246099 2608 catalog_manager.cc:5495] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 reported cstate change: term changed from 0 to 1, leader changed from <none> to 881c355127524514b175bb4a19aa8693 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "881c355127524514b175bb4a19aa8693" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } health_report { overall_health: UNKNOWN } } }
W20260620 19:15:17.246608 2935 tablet.cc:2369] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:15:17.268231 3080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.266159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55934 (local address 127.19.66.194:41829)
0620 19:15:17.266220 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:17.266222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:17.266229 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:15:17.266260 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:17.266260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:17.266278 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:17.266336 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:15:17.266418 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.266873 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.267608 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:17.267783 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:17.267825 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:17.267917 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:15:17.268011 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:15:17.268064 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:17.268122 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:15:17.268137 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:15:17.269902 3022 raft_consensus.cc:1261] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Refusing update from remote peer 881c355127524514b175bb4a19aa8693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:15:17.269923 2754 raft_consensus.cc:1261] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Refusing update from remote peer 881c355127524514b175bb4a19aa8693: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:15:17.270201 3090 consensus_queue.cc:1035] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:15:17.270337 3092 consensus_queue.cc:1035] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:15:17.272934 3096 mvcc.cc:205] Tried to move back new op lower bound from 7299002029135204352 to 7299002029037088768. Current Snapshot: MvccSnapshot[applied={T|T < 7299002029135204352}]
I20260620 19:15:17.274428 2608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:15:17.274509 2608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:15:17.275161 2608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Sending DeleteTablet for 3 replicas of tablet feae23a0ac2a4822b5a4700512820c41
I20260620 19:15:17.275488 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2671
I20260620 19:15:17.275497 2734 tablet_service.cc:1514] Processing DeleteTablet for tablet feae23a0ac2a4822b5a4700512820c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:37504
I20260620 19:15:17.275955 2868 tablet_service.cc:1514] Processing DeleteTablet for tablet feae23a0ac2a4822b5a4700512820c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55930
I20260620 19:15:17.276158 3105 tablet_replica.cc:330] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: stopping tablet replica
I20260620 19:15:17.276247 3105 raft_consensus.cc:2229] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:15:17.276322 3105 raft_consensus.cc:2258] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:15:17.276737 3002 tablet_service.cc:1514] Processing DeleteTablet for tablet feae23a0ac2a4822b5a4700512820c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39090
I20260620 19:15:17.277007 3106 tablet_replica.cc:330] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: stopping tablet replica
I20260620 19:15:17.277129 3106 raft_consensus.cc:2229] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:15:17.277283 3106 raft_consensus.cc:2258] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:15:17.280437 2596 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:37803 (error 108)
I20260620 19:15:17.280691 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2805
I20260620 19:15:17.281080 3106 ts_tablet_manager.cc:1905] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:17.281347 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.280756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:37803 (local address 127.0.0.1:37516)
0620 19:15:17.281270 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.281314 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
W20260620 19:15:17.281474 2596 catalog_manager.cc:4584] TS 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet feae23a0ac2a4822b5a4700512820c41: Network error: Client connection negotiation failed: client connection to 127.19.66.193:37803: connect: Connection refused (error 111)
I20260620 19:15:17.282600 3106 ts_tablet_manager.cc:1918] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:17.282658 3106 log.cc:1196] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/feae23a0ac2a4822b5a4700512820c41
I20260620 19:15:17.282919 3106 ts_tablet_manager.cc:1939] T feae23a0ac2a4822b5a4700512820c41 P c59822ff37a34691b98a99ea1114ef68: Deleting consensus metadata
I20260620 19:15:17.283273 2595 catalog_manager.cc:4841] TS c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): tablet feae23a0ac2a4822b5a4700512820c41 (table test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]) successfully deleted
I20260620 19:15:17.285511 3076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:17.285407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41829 (local address 127.0.0.1:55942)
0620 19:15:17.285456 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:17.285484 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.194:41829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:15:17.285632 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2939
I20260620 19:15:17.290236 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 2577
I20260620 19:15:37.294864 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--webserver_interface=127.19.66.254
--webserver_port=35257
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:37.375610 3108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:37.375804 3108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:37.375834 3108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:37.376992 3108 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:15:37.377029 3108 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:15:37.377041 3108 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:15:37.377051 3108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:15:37.377074 3108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:37.377091 3108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:37.377105 3108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:15:37.377121 3108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:15:37.378572 3108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:37.378762 3108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:37.378944 3108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:37.386919 3114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.387080 3108 server_base.cc:1034] running on GCE node
W20260620 19:15:37.386973 3117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.386973 3115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.387413 3108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:37.387706 3108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:37.388860 3108 hybrid_clock.cc:648] HybridClock initialized: now 1781982937388832 us; error 25 us; skew 500 ppm
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:37.391235 3108 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:15:37.391650 3108 webserver.cc:460] Webserver started at http://127.19.66.254:35257/ using document root <none> and password file <none>
I20260620 19:15:37.391865 3108 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:37.391930 3108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:37.393280 3108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.393858 3124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.394006 3108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:37.394085 3108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "083ca6d1218749aca9a64c1133bfc4c3"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "dc3d17243c284a7bdf06239bb0949939"
server_key_iv: "03432133c8e0cc0f9d9b0c125c0bca55"
server_key_version: "encryptionkey@0"
I20260620 19:15:37.394465 3108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:37.427791 3108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:37.430065 3108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:37.430195 3108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:37.433996 3108 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:40905
I20260620 19:15:37.434046 3176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:40905 every 8 connection(s)
I20260620 19:15:37.434398 3108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:15:37.435850 3177 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:15:37.436488 3177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap starting.
I20260620 19:15:37.440001 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3108
I20260620 19:15:37.440305 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:37803
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=44551
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:37.445219 3177 log.cc:824] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:37.447021 3177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:37.447284 3177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap complete.
I20260620 19:15:37.449394 3177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:37.449622 3177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.449786 3177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:37.449858 3177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:15:37.449910 3177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:15:37.449954 3177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 1 FOLLOWER]: Advancing to term 2
I20260620 19:15:37.450935 3177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:37.451025 3177 leader_election.cc:302] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 083ca6d1218749aca9a64c1133bfc4c3; no voters:
I20260620 19:15:37.451171 3177 leader_election.cc:288] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 19:15:37.451236 3181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 19:15:37.451365 3181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 LEADER]: Becoming Leader. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Running, Role: LEADER
I20260620 19:15:37.451436 3177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:15:37.451545 3181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:37.451841 3183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 083ca6d1218749aca9a64c1133bfc4c3. Latest consensus state: current_term: 2 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:37.451938 3183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:37.452092 3182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:37.452142 3182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:37.452564 3190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:15:37.453502 3190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:15:37.453594 3190 catalog_manager.cc:1259] Loaded cluster ID: 8df66e70df9f420591be8cae72a89bc7
I20260620 19:15:37.453615 3190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:15:37.454774 3190 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:15:37.454962 3190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Loaded TSK: 0
I20260620 19:15:37.471925 3190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Generated new TSK 1
I20260620 19:15:37.472033 3190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:15:37.526391 3179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:37.526607 3179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:37.526631 3179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:37.527969 3179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:37.528028 3179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:37.528055 3179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:15:37.529608 3179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:37803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=44551
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:37.529843 3179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:37.530074 3179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:37.536199 3205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.536208 3207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.536227 3179 server_base.cc:1034] running on GCE node
W20260620 19:15:37.536499 3204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.536763 3179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:37.537082 3179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:37.538283 3179 hybrid_clock.cc:648] HybridClock initialized: now 1781982937538264 us; error 35 us; skew 500 ppm
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:37.540954 3179 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:15:37.541384 3179 webserver.cc:460] Webserver started at http://127.19.66.193:44551/ using document root <none> and password file <none>
I20260620 19:15:37.541592 3179 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:37.541656 3179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:37.543037 3179 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.543681 3214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.543859 3179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:37.543951 3179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "286bf1a9b21b492784c4fcbdbed29002"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "96ab29af1744f4c253beb3ffeb138365"
server_key_iv: "f63cd563b54e9fa791b91c510ae1c794"
server_key_version: "encryptionkey@0"
I20260620 19:15:37.544291 3179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:37.551467 3179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:37.553689 3179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:37.553825 3179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:37.554031 3179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:37.554446 3221 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:15:37.556314 3179 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:15:37.556354 3179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:15:37.556387 3179 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:15:37.557056 3179 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:15:37.557088 3179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.557178 3221 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap starting.
I20260620 19:15:37.563654 3179 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:37803
I20260620 19:15:37.563781 3328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:37803 every 8 connection(s)
I20260620 19:15:37.564065 3179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:15:37.565197 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3179
I20260620 19:15:37.565472 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:41829
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=38189
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:37.567927 3221 log.cc:824] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:37.570812 3221 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:37.571103 3221 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap complete.
I20260620 19:15:37.572161 3221 ts_tablet_manager.cc:1397] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260620 19:15:37.573638 3221 raft_consensus.cc:348] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:37.574349 3221 raft_consensus.cc:729] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bf1a9b21b492784c4fcbdbed29002, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.574513 3221 consensus_queue.cc:260] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:37.574771 3221 ts_tablet_manager.cc:1428] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:37.579762 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.564695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:47781)
0620 19:15:37.564893 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.564944 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.565548 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.568673 (+ 3125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.568678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.569060 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:37.569352 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.569359 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.571204 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.571205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.572523 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.572526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.573487 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.573488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.573636 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.574229 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:37.574239 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:37.575947 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:37.578019 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.578026 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.578028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.578331 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.578589 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.578589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.578590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.578719 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.579139 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:37.579143 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:37.579288 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.579384 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.579541 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260620 19:15:37.580225 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.564807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:47781 (local address 127.19.66.254:40905)
0620 19:15:37.568218 (+ 3411us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.568228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.568275 (+ 47us) server_negotiation.cc:407] Connection header received
0620 19:15:37.568409 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.568416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.568472 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.568617 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:37.569505 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.571022 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.572692 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.573377 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.573941 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.576053 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:37.576058 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:37.576058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:37.576070 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:37.577931 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.578412 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.578415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.578415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.578518 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.578790 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.578791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.578791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.579072 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:37.579154 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.579926 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.580050 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3355,"thread_start_us":101,"threads_started":1}
I20260620 19:15:37.580755 3329 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:37.580876 3329 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:37.581053 3329 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:37.581769 3141 ts_manager.cc:194] Registered new tserver with Master: 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803)
I20260620 19:15:37.582998 3141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:47781
I20260620 19:15:37.583999 3329 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
I20260620 19:15:37.588814 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.582374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:37803 (local address 127.0.0.1:55108)
0620 19:15:37.584955 (+ 2581us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.584993 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.585056 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.585317 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.585320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.585323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:37.585386 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.585388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.586565 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.586566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.587825 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.587828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.588498 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.588499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.588577 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.588591 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.588658 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.588738 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2550,"thread_start_us":82,"threads_started":1}
I20260620 19:15:37.588896 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.582594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.19.66.193:37803)
0620 19:15:37.585169 (+ 2575us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.585173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.585182 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:15:37.585211 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.585213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.585226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.585290 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:37.585455 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.586482 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.587932 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.588423 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.588483 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.588520 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.588709 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.588756 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2545,"thread_start_us":59,"threads_started":1}
I20260620 19:15:37.589282 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet feae23a0ac2a4822b5a4700512820c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:55108
I20260620 19:15:37.589836 3338 tablet_replica.cc:330] stopping tablet replica
I20260620 19:15:37.589932 3338 raft_consensus.cc:2229] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:15:37.590000 3338 raft_consensus.cc:2258] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:15:37.590276 3338 ts_tablet_manager.cc:1905] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:37.592237 3338 ts_tablet_manager.cc:1918] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:37.592303 3338 log.cc:1196] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal/wals/feae23a0ac2a4822b5a4700512820c41
I20260620 19:15:37.592492 3338 ts_tablet_manager.cc:1939] T feae23a0ac2a4822b5a4700512820c41 P 286bf1a9b21b492784c4fcbdbed29002: Deleting consensus metadata
I20260620 19:15:37.592942 3128 catalog_manager.cc:4841] TS 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803): tablet feae23a0ac2a4822b5a4700512820c41 (table test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]) successfully deleted
W20260620 19:15:37.650218 3333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:37.650429 3333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:37.650452 3333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:37.651674 3333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:37.651747 3333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:37.651773 3333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:15:37.653343 3333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:41829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=38189
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:37.653604 3333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:37.653821 3333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:37.658444 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.658458 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.658504 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.658704 3333 server_base.cc:1034] running on GCE node
I20260620 19:15:37.658893 3333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:37.659109 3333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:37.660270 3333 hybrid_clock.cc:648] HybridClock initialized: now 1781982937660252 us; error 29 us; skew 500 ppm
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:37.666208 3333 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:15:37.666658 3333 webserver.cc:460] Webserver started at http://127.19.66.194:38189/ using document root <none> and password file <none>
I20260620 19:15:37.666879 3333 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:37.666944 3333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:37.668227 3333 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:15:37.668819 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.668987 3333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:37.669070 3333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "881c355127524514b175bb4a19aa8693"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "da84918022edc6df2ca3372154fff155"
server_key_iv: "070ff827225f2c1bd6aaa5244576adde"
server_key_version: "encryptionkey@0"
I20260620 19:15:37.669430 3333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:37.678028 3333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:37.679740 3333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:37.679876 3333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:37.680074 3333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:37.680456 3360 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:15:37.681834 3333 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:15:37.681876 3333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:15:37.681908 3333 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:15:37.682523 3333 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:15:37.682555 3333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:15:37.682600 3360 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Bootstrap starting.
I20260620 19:15:37.688421 3333 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:41829
I20260620 19:15:37.688482 3467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:41829 every 8 connection(s)
I20260620 19:15:37.688786 3333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:15:37.690143 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3333
I20260620 19:15:37.690447 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:33227
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=40027
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:37.692091 3360 log.cc:824] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:37.694367 3360 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:37.694665 3360 tablet_bootstrap.cc:492] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Bootstrap complete.
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:37.695685 3360 ts_tablet_manager.cc:1397] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260620 19:15:37.696781 3360 raft_consensus.cc:348] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:37.697098 3360 raft_consensus.cc:729] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881c355127524514b175bb4a19aa8693, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.697270 3360 consensus_queue.cc:260] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } }
I20260620 19:15:37.697531 3360 ts_tablet_manager.cc:1428] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 19:15:37.698836 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.689534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:47805 (local address 127.19.66.254:40905)
0620 19:15:37.689606 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.689608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.689795 (+ 187us) server_negotiation.cc:407] Connection header received
0620 19:15:37.690273 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.690275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.690286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.690316 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:37.691078 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.691573 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.692970 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.693233 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.693294 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.695879 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:37.695881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:37.695882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:37.695891 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:37.697508 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.697894 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.697895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.697896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.697960 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.698196 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.698197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.698197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.698348 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:37.698438 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.698651 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.698743 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 19:15:37.698868 3470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.689474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:47805)
0620 19:15:37.689673 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.689713 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.690166 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.690453 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.690456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.690786 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:37.690987 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.690992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.691670 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.691671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.692880 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.692892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.693334 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.693336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.693487 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.694053 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:37.694063 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:37.695743 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:37.697601 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.697606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.697608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.697818 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.698032 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.698033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.698033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.698136 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.698425 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:37.698428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:37.698537 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.698613 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.698734 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260620 19:15:37.699362 3468 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:37.699448 3468 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:37.699605 3468 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:37.700106 3141 ts_manager.cc:194] Registered new tserver with Master: 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
I20260620 19:15:37.700706 3141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:47805
I20260620 19:15:37.701481 3468 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.194@KRBTEST.COM
I20260620 19:15:37.707646 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.700448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.19.66.194:41829)
0620 19:15:37.700586 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.700590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.700598 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:15:37.700622 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.700624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.700634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.700690 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:37.701191 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.702257 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.702860 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.703301 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.703360 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.705234 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:37.705240 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:37.705241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:37.705247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:37.706606 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.706934 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.706937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.706937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.706990 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.707144 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.707145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.707145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.707275 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:37.707341 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.707489 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.707564 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260620 19:15:37.707644 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.700390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41829 (local address 127.0.0.1:39226)
0620 19:15:37.700438 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.700447 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.700483 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.700702 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.700703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.701080 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:37.701119 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.701122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.702327 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.702328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.702798 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.702800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.703365 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.703366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.703403 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.703824 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:37.703829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:37.705159 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:37.706682 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.706689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.706691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.706862 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.707041 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.707042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.707042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.707096 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.707332 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:37.707338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:37.707384 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.707451 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.707545 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:15:37.708250 3402 tablet_service.cc:1514] Processing DeleteTablet for tablet feae23a0ac2a4822b5a4700512820c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39226
I20260620 19:15:37.708803 3475 tablet_replica.cc:330] stopping tablet replica
I20260620 19:15:37.708935 3475 raft_consensus.cc:2229] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:15:37.709003 3475 raft_consensus.cc:2258] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:15:37.709275 3475 ts_tablet_manager.cc:1905] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:37.710942 3475 ts_tablet_manager.cc:1918] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:37.711001 3475 log.cc:1196] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/feae23a0ac2a4822b5a4700512820c41
I20260620 19:15:37.711174 3475 ts_tablet_manager.cc:1939] T feae23a0ac2a4822b5a4700512820c41 P 881c355127524514b175bb4a19aa8693: Deleting consensus metadata
I20260620 19:15:37.711602 3125 catalog_manager.cc:4841] TS 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): tablet feae23a0ac2a4822b5a4700512820c41 (table test-table [id=bd0aea8ee3b544e6b6178a8543985cb2]) successfully deleted
W20260620 19:15:37.774099 3471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:37.774313 3471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:37.774339 3471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:37.775545 3471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:37.775588 3471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:37.775614 3471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:15:37.777096 3471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:33227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=40027
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:37.777323 3471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:37.777539 3471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:37.782282 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.782277 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:37.782316 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:37.782577 3471 server_base.cc:1034] running on GCE node
I20260620 19:15:37.782790 3471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:37.783022 3471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:37.784165 3471 hybrid_clock.cc:648] HybridClock initialized: now 1781982937784151 us; error 29 us; skew 500 ppm
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:37.786702 3471 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:15:37.787147 3471 webserver.cc:460] Webserver started at http://127.19.66.195:40027/ using document root <none> and password file <none>
I20260620 19:15:37.787358 3471 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:37.787424 3471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:37.788757 3471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.789355 3490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.789507 3471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.789592 3471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "c59822ff37a34691b98a99ea1114ef68"
format_stamp: "Formatted at 2026-06-20 19:15:17 on dist-test-slave-nq8r"
server_key: "bf6c1f223fab8b3c30271ea1aef29487"
server_key_iv: "c4b230dc08974b7141dcb13b9d683e46"
server_key_version: "encryptionkey@0"
I20260620 19:15:37.789939 3471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:37.821158 3471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:37.822875 3471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:37.823014 3471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:37.823222 3471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:37.823499 3471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:37.823531 3471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.823560 3471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:37.823581 3471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.828771 3471 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33227
I20260620 19:15:37.828819 3603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33227 every 8 connection(s)
I20260620 19:15:37.829162 3471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:15:37 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982937, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:15:37.836130 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3471
I20260620 19:15:37.837536 3332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.829800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:45511 (local address 127.19.66.254:40905)
0620 19:15:37.829876 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.829879 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.830034 (+ 155us) server_negotiation.cc:407] Connection header received
0620 19:15:37.830496 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.830498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.830507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.830552 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:37.831422 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.831896 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.832933 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.833120 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.833171 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.835023 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:37.835025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:37.835025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:37.835032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:37.836235 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.836618 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.836619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.836620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.836669 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.836945 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.836946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.836946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.837040 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:37.837086 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.837379 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.837453 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:15:37.837747 3606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.829786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:45511)
0620 19:15:37.829920 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.829955 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.830422 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.830615 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.830618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.831159 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:37.831346 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.831351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.831956 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.831956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.832837 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.832839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.833178 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.833178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.833274 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.833702 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:37.833710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:37.834947 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:37.836310 (+ 1363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.836316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.836318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.836557 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.836726 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:37.836727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:37.836727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:37.836890 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:37.837094 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:37.837100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:37.837258 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.837370 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.837531 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260620 19:15:37.838191 3604 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40905
I20260620 19:15:37.838297 3604 heartbeater.cc:464] Registering TS with master...
I20260620 19:15:37.838472 3604 heartbeater.cc:511] Master 127.19.66.254:40905 requested a full tablet report, sending...
I20260620 19:15:37.838848 3141 ts_manager.cc:194] Registered new tserver with Master: c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227)
I20260620 19:15:37.839203 3141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:45511
I20260620 19:15:37.842381 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.836503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.19.66.254:40905)
0620 19:15:37.836607 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.836609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.836733 (+ 124us) server_negotiation.cc:407] Connection header received
0620 19:15:37.836799 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.836800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.836808 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.836869 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:37.837499 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.838352 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.839171 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.839468 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.839524 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.840517 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:37.840518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:37.840518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:37.840524 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:37.841595 (+ 1071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.841864 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.841865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.841865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.841920 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:37.842037 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:37.842038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:37.842038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:37.842118 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:37.842229 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.842253 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.842307 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260620 19:15:37.842952 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:15:37.845273 3608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.843241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36390 (local address 127.19.66.254:40905)
0620 19:15:37.843294 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.843295 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.843355 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:15:37.843436 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.843436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.843463 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.843509 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:15:37.843576 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.844004 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.844667 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.844839 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.844933 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.845060 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:15:37.845145 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:15:37.845196 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.845208 (+ 12us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.845223 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:15:37.845531 3141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36390:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:15:37.847108 3141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:15:37.848992 3402 tablet_service.cc:1467] Processing CreateTablet for tablet d98c609dc52a42cdad8964c3a486f065 (DEFAULT_TABLE table=test-table [id=af952278b1394a729f226e09dffe585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:37.849020 3252 tablet_service.cc:1467] Processing CreateTablet for tablet d98c609dc52a42cdad8964c3a486f065 (DEFAULT_TABLE table=test-table [id=af952278b1394a729f226e09dffe585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:37.849187 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98c609dc52a42cdad8964c3a486f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:37.849188 3402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98c609dc52a42cdad8964c3a486f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:37.850544 3221 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap starting.
I20260620 19:15:37.850639 3360 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Bootstrap starting.
I20260620 19:15:37.850905 3360 tablet_bootstrap.cc:654] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:37.850919 3221 tablet_bootstrap.cc:654] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:37.851440 3221 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: No bootstrap required, opened a new log
I20260620 19:15:37.851491 3221 ts_tablet_manager.cc:1397] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.851641 3221 raft_consensus.cc:348] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.851713 3360 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: No bootstrap required, opened a new log
I20260620 19:15:37.851702 3221 raft_consensus.cc:374] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:37.851753 3360 ts_tablet_manager.cc:1397] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:37.851755 3221 raft_consensus.cc:729] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bf1a9b21b492784c4fcbdbed29002, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.851799 3221 consensus_queue.cc:260] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.851883 3221 ts_tablet_manager.cc:1428] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:37.851883 3360 raft_consensus.cc:348] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.851954 3360 raft_consensus.cc:374] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:37.851996 3360 raft_consensus.cc:729] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881c355127524514b175bb4a19aa8693, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.852041 3360 consensus_queue.cc:260] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.852124 3360 ts_tablet_manager.cc:1428] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:37.852881 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.848737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33227 (local address 127.0.0.1:50514)
0620 19:15:37.848765 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.848774 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.848803 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.849222 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.849222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.849228 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:37.849267 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.849268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.850487 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.850488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.851564 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.851567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.852657 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.852658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.852706 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.852709 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.852760 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.852797 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 19:15:37.852944 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.848877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.19.66.195:33227)
0620 19:15:37.849047 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.849052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.849062 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:15:37.849096 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.849097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.849113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.849331 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:37.849412 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.850389 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.851685 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.852557 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.852610 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.852660 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.852773 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.852812 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260620 19:15:37.853519 3538 tablet_service.cc:1467] Processing CreateTablet for tablet d98c609dc52a42cdad8964c3a486f065 (DEFAULT_TABLE table=test-table [id=af952278b1394a729f226e09dffe585c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:15:37.853816 3538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d98c609dc52a42cdad8964c3a486f065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:15:37.856470 3613 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Bootstrap starting.
I20260620 19:15:37.857028 3613 tablet_bootstrap.cc:654] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Neither blocks nor log segments found. Creating new log.
I20260620 19:15:37.857537 3613 log.cc:824] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:37.858232 3613 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: No bootstrap required, opened a new log
I20260620 19:15:37.858289 3613 ts_tablet_manager.cc:1397] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:15:37.859580 3613 raft_consensus.cc:348] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.859719 3613 raft_consensus.cc:374] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:15:37.859750 3613 raft_consensus.cc:729] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59822ff37a34691b98a99ea1114ef68, State: Initialized, Role: FOLLOWER
I20260620 19:15:37.859861 3613 consensus_queue.cc:260] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.860119 3613 ts_tablet_manager.cc:1428] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:15:37.860131 3604 heartbeater.cc:503] Master 127.19.66.254:40905 was elected leader, sending a full tablet report...
I20260620 19:15:37.928778 3335 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:37.928934 3335 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.929294 3335 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
I20260620 19:15:37.931893 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.929449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33227 (local address 127.19.66.193:59331)
0620 19:15:37.929502 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.929512 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.929551 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.929716 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.929717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.929719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:37.929784 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.929785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.930375 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.930376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.931134 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.931136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.931660 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.931661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.931737 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.931740 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.931791 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.931827 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":4}
I20260620 19:15:37.931933 3610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.929522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:59331 (local address 127.19.66.195:33227)
0620 19:15:37.929600 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.929603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.929613 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:15:37.929642 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.929643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.929650 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.929704 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:37.929839 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.930309 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.931217 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.931579 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.931638 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.931647 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.931816 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.931863 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 19:15:37.932360 3558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98c609dc52a42cdad8964c3a486f065" candidate_uuid: "286bf1a9b21b492784c4fcbdbed29002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59822ff37a34691b98a99ea1114ef68" is_pre_election: true
I20260620 19:15:37.932542 3558 raft_consensus.cc:2454] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286bf1a9b21b492784c4fcbdbed29002 in term 0.
I20260620 19:15:37.932626 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.929530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:35573 (local address 127.19.66.194:41829)
0620 19:15:37.929586 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.929588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.929699 (+ 111us) server_negotiation.cc:407] Connection header received
0620 19:15:37.929746 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.929747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.929771 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.929828 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:15:37.930228 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.930746 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.931670 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.932200 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.932384 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.932472 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.932546 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.932577 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:15:37.932665 3616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.929474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41829 (local address 127.19.66.193:35573)
0620 19:15:37.929647 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.929654 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:37.929693 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:37.929841 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:37.929841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:37.929843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:15:37.930162 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.930164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.930828 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.930829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.931592 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:37.931594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.932399 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.932400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:37.932477 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.932481 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:15:37.932538 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:15:37.932577 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":120,"threads_started":1}
I20260620 19:15:37.932786 3217 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002, c59822ff37a34691b98a99ea1114ef68; no voters:
I20260620 19:15:37.932924 3335 raft_consensus.cc:2781] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:15:37.932972 3335 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:15:37.932994 3335 raft_consensus.cc:3037] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:37.932978 3422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98c609dc52a42cdad8964c3a486f065" candidate_uuid: "286bf1a9b21b492784c4fcbdbed29002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881c355127524514b175bb4a19aa8693" is_pre_election: true
I20260620 19:15:37.933120 3422 raft_consensus.cc:2454] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286bf1a9b21b492784c4fcbdbed29002 in term 0.
I20260620 19:15:37.933723 3335 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.933842 3335 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 1 election: Requested vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
I20260620 19:15:37.933984 3558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98c609dc52a42cdad8964c3a486f065" candidate_uuid: "286bf1a9b21b492784c4fcbdbed29002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59822ff37a34691b98a99ea1114ef68"
I20260620 19:15:37.933992 3422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d98c609dc52a42cdad8964c3a486f065" candidate_uuid: "286bf1a9b21b492784c4fcbdbed29002" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "881c355127524514b175bb4a19aa8693"
I20260620 19:15:37.934051 3422 raft_consensus.cc:3037] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:37.934049 3558 raft_consensus.cc:3037] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:15:37.934690 3422 raft_consensus.cc:2454] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286bf1a9b21b492784c4fcbdbed29002 in term 1.
I20260620 19:15:37.934731 3558 raft_consensus.cc:2454] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286bf1a9b21b492784c4fcbdbed29002 in term 1.
I20260620 19:15:37.934885 3215 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002, 881c355127524514b175bb4a19aa8693; no voters:
I20260620 19:15:37.934981 3335 raft_consensus.cc:2781] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:15:37.935130 3335 raft_consensus.cc:686] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 LEADER]: Becoming Leader. State: Replica: 286bf1a9b21b492784c4fcbdbed29002, State: Running, Role: LEADER
I20260620 19:15:37.935194 3335 consensus_queue.cc:237] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:37.935880 3140 catalog_manager.cc:5495] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 reported cstate change: term changed from 0 to 1, leader changed from <none> to 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193). New cstate: current_term: 1 leader_uuid: "286bf1a9b21b492784c4fcbdbed29002" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } health_report { overall_health: UNKNOWN } } }
W20260620 19:15:37.939284 3469 tablet.cc:2369] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:15:37.944211 3337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.942113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.19.66.193:37803)
0620 19:15:37.942167 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:37.942169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:37.942177 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:15:37.942207 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:37.942208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:37.942223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:37.942283 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:15:37.942417 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.942867 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.943546 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:37.943743 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:37.943801 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:37.943930 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:15:37.944025 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:15:37.944083 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:37.944130 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:15:37.944149 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:15:37.945509 3422 raft_consensus.cc:1261] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Refusing update from remote peer 286bf1a9b21b492784c4fcbdbed29002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:15:37.945510 3558 raft_consensus.cc:1261] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Refusing update from remote peer 286bf1a9b21b492784c4fcbdbed29002: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:15:37.945817 3617 consensus_queue.cc:1035] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:15:37.945943 3335 consensus_queue.cc:1035] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:15:37.947978 3620 mvcc.cc:205] Tried to move back new op lower bound from 7299002113822711808 to 7299002113783058432. Current Snapshot: MvccSnapshot[applied={T|T < 7299002113822711808}]
I20260620 19:15:37.948187 3621 mvcc.cc:205] Tried to move back new op lower bound from 7299002113822711808 to 7299002113783058432. Current Snapshot: MvccSnapshot[applied={T|T < 7299002113822711808}]
I20260620 19:15:37.948872 3622 mvcc.cc:205] Tried to move back new op lower bound from 7299002113822711808 to 7299002113783058432. Current Snapshot: MvccSnapshot[applied={T|T < 7299002113822711808 or (T in {7299002113822711808})}]
I20260620 19:15:37.949942 3141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:15:37.950018 3141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36390:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:15:37.950639 3141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Sending DeleteTablet for 3 replicas of tablet d98c609dc52a42cdad8964c3a486f065
I20260620 19:15:37.950954 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3179
I20260620 19:15:37.950954 3538 tablet_service.cc:1514] Processing DeleteTablet for tablet d98c609dc52a42cdad8964c3a486f065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:50514
I20260620 19:15:37.951785 3402 tablet_service.cc:1514] Processing DeleteTablet for tablet d98c609dc52a42cdad8964c3a486f065 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:15:37 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:39226
I20260620 19:15:37.951871 3475 tablet_replica.cc:330] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: stopping tablet replica
I20260620 19:15:37.951920 3475 raft_consensus.cc:2229] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:15:37.951967 3475 raft_consensus.cc:2258] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:15:37.952087 3630 tablet_replica.cc:330] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: stopping tablet replica
I20260620 19:15:37.952184 3630 raft_consensus.cc:2229] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:15:37.952280 3630 raft_consensus.cc:2258] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:15:37.952697 3630 ts_tablet_manager.cc:1905] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:37.953292 3475 ts_tablet_manager.cc:1905] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:15:37.954155 3630 ts_tablet_manager.cc:1918] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:37.954222 3630 log.cc:1196] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/d98c609dc52a42cdad8964c3a486f065
I20260620 19:15:37.954468 3630 ts_tablet_manager.cc:1939] T d98c609dc52a42cdad8964c3a486f065 P c59822ff37a34691b98a99ea1114ef68: Deleting consensus metadata
I20260620 19:15:37.954680 3475 ts_tablet_manager.cc:1918] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:15:37.954734 3475 log.cc:1196] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/d98c609dc52a42cdad8964c3a486f065
I20260620 19:15:37.954859 3127 catalog_manager.cc:4841] TS c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): tablet d98c609dc52a42cdad8964c3a486f065 (table test-table [id=af952278b1394a729f226e09dffe585c]) successfully deleted
I20260620 19:15:37.954917 3475 ts_tablet_manager.cc:1939] T d98c609dc52a42cdad8964c3a486f065 P 881c355127524514b175bb4a19aa8693: Deleting consensus metadata
I20260620 19:15:37.955231 3125 catalog_manager.cc:4841] TS 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): tablet d98c609dc52a42cdad8964c3a486f065 (table test-table [id=af952278b1394a729f226e09dffe585c]) successfully deleted
W20260620 19:15:37.956295 3128 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:37803 (error 108)
I20260620 19:15:37.956596 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3333
I20260620 19:15:37.956699 3336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:37.956583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:37803 (local address 127.0.0.1:55132)
0620 19:15:37.956632 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:37.956671 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:37803: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 19:15:37.960914 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3471
W20260620 19:15:37.960832 3128 catalog_manager.cc:4584] TS 286bf1a9b21b492784c4fcbdbed29002 (127.19.66.193:37803): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d98c609dc52a42cdad8964c3a486f065: Network error: Client connection negotiation failed: client connection to 127.19.66.193:37803: connect: Connection refused (error 111)
I20260620 19:15:37.965071 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3108
I20260620 19:15:57.969493 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--webserver_interface=127.19.66.254
--webserver_port=35257
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:58.047667 3632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:58.047856 3632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:58.047875 3632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:58.048956 3632 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 19:15:58.048986 3632 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 19:15:58.048997 3632 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 19:15:58.049007 3632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:15:58.049024 3632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:58.049036 3632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:58.049046 3632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:15:58.049062 3632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:15:58.050433 3632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:58.050623 3632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:58.050786 3632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:15:58.055488 3632 server_base.cc:1034] running on GCE node
W20260620 19:15:58.055498 3638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.055429 3637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.055421 3640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.055882 3632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:58.056151 3632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:58.057287 3632 hybrid_clock.cc:648] HybridClock initialized: now 1781982958057273 us; error 23 us; skew 500 ppm
Jun 20 19:15:58 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:58.064263 3632 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:15:58.064697 3632 webserver.cc:460] Webserver started at http://127.19.66.254:35257/ using document root <none> and password file <none>
I20260620 19:15:58.064944 3632 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:58.064996 3632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:58.066357 3632 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:15:58.067026 3647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.067175 3632 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:15:58.067253 3632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "083ca6d1218749aca9a64c1133bfc4c3"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "dc3d17243c284a7bdf06239bb0949939"
server_key_iv: "03432133c8e0cc0f9d9b0c125c0bca55"
server_key_version: "encryptionkey@0"
I20260620 19:15:58.067616 3632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:58.085655 3632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:58.087654 3632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:58.087790 3632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:58.091454 3632 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:40905
I20260620 19:15:58.091497 3699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:40905 every 8 connection(s)
I20260620 19:15:58.091856 3632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:15:58.093336 3700 sys_catalog.cc:263] Verifying existing consensus state
I20260620 19:15:58.093917 3700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap starting.
I20260620 19:15:58.094082 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3632
I20260620 19:15:58.094323 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:37803
--local_ip_for_outbound_sockets=127.19.66.193
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=44551
--webserver_interface=127.19.66.193
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:58.102099 3700 log.cc:824] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:58.104287 3700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:58.104550 3700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Bootstrap complete.
I20260620 19:15:58.106582 3700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:58.106794 3700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Initialized, Role: FOLLOWER
I20260620 19:15:58.106940 3700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:58.107019 3700 raft_consensus.cc:388] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:15:58.107064 3700 raft_consensus.cc:482] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:15:58.107110 3700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 2 FOLLOWER]: Advancing to term 3
I20260620 19:15:58.108052 3700 raft_consensus.cc:504] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:58.108142 3700 leader_election.cc:302] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 083ca6d1218749aca9a64c1133bfc4c3; no voters:
I20260620 19:15:58.108295 3700 leader_election.cc:288] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 19:15:58.108346 3704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 19:15:58.108461 3704 raft_consensus.cc:686] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [term 3 LEADER]: Becoming Leader. State: Replica: 083ca6d1218749aca9a64c1133bfc4c3, State: Running, Role: LEADER
I20260620 19:15:58.108542 3704 consensus_queue.cc:237] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } }
I20260620 19:15:58.108565 3700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:15:58.108798 3704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 083ca6d1218749aca9a64c1133bfc4c3. Latest consensus state: current_term: 3 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:58.108803 3705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "083ca6d1218749aca9a64c1133bfc4c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "083ca6d1218749aca9a64c1133bfc4c3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40905 } } }
I20260620 19:15:58.108901 3704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:58.108927 3705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:15:58.109155 3708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:15:58.109989 3708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:15:58.110090 3708 catalog_manager.cc:1259] Loaded cluster ID: 8df66e70df9f420591be8cae72a89bc7
I20260620 19:15:58.110118 3708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:15:58.111225 3708 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:15:58.111405 3708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Loaded TSK: 0
I20260620 19:15:58.111454 3708 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 083ca6d1218749aca9a64c1133bfc4c3: Loaded TSK: 1
F20260620 19:15:58.111667 3708 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1762b7f9ad google::LogMessage::Fail() at ??:0
@ 0x7f1762b8288a google::LogMessage::SendToLog() at ??:0
@ 0x7f1762b7f4de google::LogMessage::Flush() at ??:0
@ 0x7f1762b80a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f17652ec104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f17652e9d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f176aadd1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f176aadd48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f176aadd594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f176aaa3c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f176aaadad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1764a4d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1764a42954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f17658a66db start_thread at ??:0
@ 0x7f176167d61f clone at ??:0
W20260620 19:15:58.193833 3702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:58.194018 3702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:58.194047 3702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:58.195156 3702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:58.195192 3702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:58.195216 3702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:15:58.196607 3702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:37803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=44551
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:58.196815 3702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:58.197048 3702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:58.201694 3744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.201663 3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.201673 3742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.201903 3702 server_base.cc:1034] running on GCE node
I20260620 19:15:58.202076 3702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:58.202301 3702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:58.203433 3702 hybrid_clock.cc:648] HybridClock initialized: now 1781982958203421 us; error 32 us; skew 500 ppm
Jun 20 19:15:58 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:58.205739 3702 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:15:58.206146 3702 webserver.cc:460] Webserver started at http://127.19.66.193:44551/ using document root <none> and password file <none>
I20260620 19:15:58.206342 3702 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:58.206389 3702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:58.207684 3702 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:58.208312 3751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.208457 3702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:15:58.208542 3702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "286bf1a9b21b492784c4fcbdbed29002"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "96ab29af1744f4c253beb3ffeb138365"
server_key_iv: "f63cd563b54e9fa791b91c510ae1c794"
server_key_version: "encryptionkey@0"
I20260620 19:15:58.208873 3702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:58.225548 3702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:58.227641 3702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:58.227758 3702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:58.227958 3702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:58.228363 3758 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 19:15:58.230293 3702 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 19:15:58.230336 3702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 19:15:58.230367 3702 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 19:15:58.230971 3702 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 19:15:58.231003 3702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 19:15:58.231101 3758 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap starting.
I20260620 19:15:58.237422 3702 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:37803
I20260620 19:15:58.237794 3702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:15:58.238270 3865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:37803 every 8 connection(s)
I20260620 19:15:58.241693 3758 log.cc:824] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Log is configured to *not* fsync() on all Append() calls
I20260620 19:15:58.243811 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3702
I20260620 19:15:58.244047 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:41829
--local_ip_for_outbound_sockets=127.19.66.194
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=38189
--webserver_interface=127.19.66.194
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:15:58.244365 3758 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 19:15:58.244647 3758 tablet_bootstrap.cc:492] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Bootstrap complete.
I20260620 19:15:58.245729 3758 ts_tablet_manager.cc:1397] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260620 19:15:58.247129 3758 raft_consensus.cc:348] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:58.247467 3758 raft_consensus.cc:729] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286bf1a9b21b492784c4fcbdbed29002, State: Initialized, Role: FOLLOWER
I20260620 19:15:58.247640 3758 consensus_queue.cc:260] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:58.247917 3758 ts_tablet_manager.cc:1428] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260620 19:15:58.297814 3868 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:15:58.238447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:38579)
0620 19:15:58.238607 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.238648 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:58.239088 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:58.297640 (+ 58552us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
W20260620 19:15:58.298055 3866 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:40905 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:40905: Client connection negotiation failed: client connection to 127.19.66.254:40905: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 19:15:58.298391 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.298278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:54031)
0620 19:15:58.298324 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.298371 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:15:58.298704 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.298626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:47329)
0620 19:15:58.298662 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.298684 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:15:58.298811 3866 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 19:15:58.324666 3870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:58.324869 3870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:58.324901 3870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:58.326021 3870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:58.326056 3870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:58.326082 3870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:15:58.327461 3870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:41829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=38189
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:58.327667 3870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:58.327863 3870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:58.332497 3877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.332515 3879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.332546 3870 server_base.cc:1034] running on GCE node
W20260620 19:15:58.332530 3876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.332914 3870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:58.333164 3870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:58.334298 3870 hybrid_clock.cc:648] HybridClock initialized: now 1781982958334287 us; error 29 us; skew 500 ppm
Jun 20 19:15:58 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:58.336581 3870 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:15:58.337023 3870 webserver.cc:460] Webserver started at http://127.19.66.194:38189/ using document root <none> and password file <none>
I20260620 19:15:58.337211 3870 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:58.337250 3870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:58.338423 3870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:58.338905 3886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.339066 3870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.339144 3870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "881c355127524514b175bb4a19aa8693"
format_stamp: "Formatted at 2026-06-20 19:15:16 on dist-test-slave-nq8r"
server_key: "da84918022edc6df2ca3372154fff155"
server_key_iv: "070ff827225f2c1bd6aaa5244576adde"
server_key_version: "encryptionkey@0"
I20260620 19:15:58.339460 3870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:58.352684 3870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:58.354453 3870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:58.354581 3870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:58.354768 3870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:58.355043 3870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:58.355078 3870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.355105 3870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:58.355129 3870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.360532 3870 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:41829
I20260620 19:15:58.360570 3999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:41829 every 8 connection(s)
I20260620 19:15:58.360917 3870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:15:58.361850 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.361504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:41385)
0620 19:15:58.361649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.361717 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
W20260620 19:15:58.362228 4000 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:40905 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:40905: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
I20260620 19:15:58.362501 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.362417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:36679)
0620 19:15:58.362459 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.362485 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 19:15:58.362738 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.362676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:47471)
0620 19:15:58.362708 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.362726 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:15:58.362831 4000 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:15:58.368538 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 3870
I20260620 19:15:58.368779 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:33227
--local_ip_for_outbound_sockets=127.19.66.195
--tserver_master_addrs=127.19.66.254:40905
--webserver_port=40027
--webserver_interface=127.19.66.195
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:15:58.447679 4003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:15:58.447873 4003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:15:58.447896 4003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:15:58.449055 4003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:15:58.449095 4003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:15:58.449126 4003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:15:58.450532 4003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:33227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=40027
--tserver_master_addrs=127.19.66.254:40905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:15:58.450742 4003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:15:58.450943 4003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:15:58.455369 4009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:15:58.455410 4008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.455451 4003 server_base.cc:1034] running on GCE node
W20260620 19:15:58.455408 4011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:15:58.455717 4003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:15:58.455919 4003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:15:58.457044 4003 hybrid_clock.cc:648] HybridClock initialized: now 1781982958457032 us; error 29 us; skew 500 ppm
Jun 20 19:15:58 dist-test-slave-nq8r krb5kdc[2564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:15:58.459263 4003 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:15:58.459654 4003 webserver.cc:460] Webserver started at http://127.19.66.195:40027/ using document root <none> and password file <none>
I20260620 19:15:58.459817 4003 fs_manager.cc:359] Metadata directory not provided
I20260620 19:15:58.459854 4003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:15:58.461092 4003 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:15:58.461591 4018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.461735 4003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.461812 4003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "c59822ff37a34691b98a99ea1114ef68"
format_stamp: "Formatted at 2026-06-20 19:15:17 on dist-test-slave-nq8r"
server_key: "bf6c1f223fab8b3c30271ea1aef29487"
server_key_iv: "c4b230dc08974b7141dcb13b9d683e46"
server_key_version: "encryptionkey@0"
I20260620 19:15:58.462134 4003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:15:58.501910 4003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:15:58.504089 4003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:15:58.504216 4003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:15:58.504417 4003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:15:58.504735 4003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:15:58.504771 4003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.504801 4003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:15:58.504818 4003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:15:58.510146 4003 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33227
I20260620 19:15:58.510202 4131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33227 every 8 connection(s)
I20260620 19:15:58.510530 4003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:15:58.511590 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.511163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:46071)
0620 19:15:58.511333 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.511434 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
W20260620 19:15:58.511965 4132 heartbeater.cc:644] Failed to heartbeat to 127.19.66.254:40905 (0 consecutive failures): Network error: Failed to ping master at 127.19.66.254:40905: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
I20260620 19:15:58.512224 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.512139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:52225)
0620 19:15:58.512180 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.512204 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:15:58.512526 4134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:58.512463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:48553)
0620 19:15:58.512490 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:58.512511 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 19:15:58.512617 4132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 19:15:58.513255 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4003
I20260620 19:15:59.299391 4136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.299152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:50695)
0620 19:15:59.299288 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:59.299328 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260620 19:15:59.363315 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.363121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:39535)
0620 19:15:59.363229 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:59.363267 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":53,"threads_started":1}
I20260620 19:15:59.513185 4138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.512990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:56609)
0620 19:15:59.513098 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:59.513137 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260620 19:15:59.862782 4139 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:15:59.862893 4139 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:15:59.863216 4139 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
Jun 20 19:15:59 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.194@KRBTEST.COM
Jun 20 19:15:59 dist-test-slave-nq8r krb5kdc[2564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982958, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.195@KRBTEST.COM
I20260620 19:15:59.873150 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.863365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:41829 (local address 127.19.66.193:39459)
0620 19:15:59.863624 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:59.863634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:59.863688 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:59.864391 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:59.864394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:59.864746 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:59.864963 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:59.864966 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.866020 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.866021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:59.866802 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:59.866804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.867335 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.867336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:59.867449 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:59.867974 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:59.867981 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:59.869423 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:59.871342 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:59.871351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:59.871738 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:59.871993 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:59.871994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:59.872145 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:59.872584 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:59.872590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:59.872740 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 19:15:59.872875 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:15:59.873041 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":53,"thread_start_us":91,"threads_started":1}
I20260620 19:15:59.873178 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.863554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:57335 (local address 127.19.66.195:33227)
0620 19:15:59.863694 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:59.863705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:59.863734 (+ 29us) server_negotiation.cc:407] Connection header received
0620 19:15:59.864163 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:59.864165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:59.864204 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:59.864344 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:59.865063 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.866012 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.866879 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.867269 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.867358 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:59.869780 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:59.869784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:59.869785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:59.869795 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:59.871312 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871811 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:59.871814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:59.871814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:59.871929 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:59.872205 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:59.872206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:59.872206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:59.872530 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:59.872597 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:59.872943 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 19:15:59.873070 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":43,"threads_started":1}
I20260620 19:15:59.873178 4142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.863490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:39459 (local address 127.19.66.194:41829)
0620 19:15:59.863638 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 19:15:59.863651 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 19:15:59.863681 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:15:59.864157 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:15:59.864160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:15:59.864208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:15:59.864334 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:15:59.865066 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.865930 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.866880 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.867258 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.867357 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:59.869553 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:15:59.869558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:15:59.869558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:15:59.869570 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:15:59.871269 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871811 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:59.871814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:59.871814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:59.871936 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:15:59.872205 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:15:59.872205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:15:59.872206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:15:59.872511 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:15:59.872590 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 19:15:59.872947 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 19:15:59.873070 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":54,"threads_started":1}
I20260620 19:15:59.873478 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:15:59.863337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33227 (local address 127.19.66.193:57335)
0620 19:15:59.863494 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:15:59.863505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:15:59.863563 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:15:59.864391 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:15:59.864394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:15:59.864759 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:15:59.864959 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:59.864965 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.866093 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.866094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:59.866802 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:15:59.866805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:15:59.867337 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:15:59.867338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:15:59.867450 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:15:59.868001 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:15:59.868006 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:15:59.869693 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:15:59.871390 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:59.871391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:59.871738 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:59.871993 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:15:59.871993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:15:59.871994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:15:59.872145 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:15:59.872586 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:15:59.872590 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:15:59.872740 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 19:15:59.872875 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 19:15:59.873103 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":79,"thread_start_us":66,"threads_started":1}
W20260620 19:15:59.874020 3752 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
W20260620 19:15:59.874203 3754 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
I20260620 19:15:59.874243 3754 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002; no voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68
I20260620 19:15:59.874339 4139 raft_consensus.cc:2726] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:16:00.300000 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:00.299868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:45181)
0620 19:16:00.299924 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:00.299961 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:16:00.363945 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:00.363730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:40895)
0620 19:16:00.363852 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:00.363888 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260620 19:16:00.513854 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:00.513640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:39695)
0620 19:16:00.513772 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:00.513814 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260620 19:16:01.300676 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:01.300446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:59739)
0620 19:16:01.300593 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:01.300631 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260620 19:16:01.364530 4147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:01.364320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:51923)
0620 19:16:01.364459 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:01.364492 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260620 19:16:01.514480 4148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:01.514272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:51721)
0620 19:16:01.514400 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:01.514437 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260620 19:16:01.606855 4149 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:01.606953 4149 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:16:01.607115 4149 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
W20260620 19:16:01.607594 3754 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
W20260620 19:16:01.607694 3752 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
I20260620 19:16:01.607762 3752 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002; no voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68
I20260620 19:16:01.607833 4149 raft_consensus.cc:2726] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:16:02.301291 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:02.301096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:35425)
0620 19:16:02.301215 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:02.301246 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260620 19:16:02.365170 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:02.364977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:35597)
0620 19:16:02.365092 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:02.365129 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260620 19:16:02.515136 4152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:02.514909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:36431)
0620 19:16:02.515051 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:02.515093 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260620 19:16:03.302055 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:03.301830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:35571)
0620 19:16:03.301970 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:03.302011 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260620 19:16:03.365840 4154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:03.365628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:38705)
0620 19:16:03.365754 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:03.365798 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
I20260620 19:16:03.515887 4155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:03.515626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:47695)
0620 19:16:03.515786 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:03.515837 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":62,"threads_started":1}
I20260620 19:16:04.302716 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:04.302515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:42399)
0620 19:16:04.302635 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:04.302668 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260620 19:16:04.366457 4157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:04.366270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:53711)
0620 19:16:04.366383 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:04.366418 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 19:16:04.516624 4158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:04.516383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:49455)
0620 19:16:04.516524 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:04.516570 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260620 19:16:04.892554 4159 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:04.892648 4159 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:16:04.892781 4159 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
W20260620 19:16:04.893131 3754 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
W20260620 19:16:04.893208 3752 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
I20260620 19:16:04.893277 3752 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002; no voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68
I20260620 19:16:04.893352 4159 raft_consensus.cc:2726] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:16:05.303404 4160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:05.303157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:60077)
0620 19:16:05.303319 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:05.303360 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260620 19:16:05.367039 4161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:05.366834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:50211)
0620 19:16:05.366957 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:05.366999 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260620 19:16:05.517220 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:05.517045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:42249)
0620 19:16:05.517145 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:05.517178 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260620 19:16:06.304117 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:06.303875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:49521)
0620 19:16:06.304028 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:06.304069 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260620 19:16:06.367643 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:06.367423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:54599)
0620 19:16:06.367537 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:06.367573 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260620 19:16:06.517871 4165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:06.517641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:42691)
0620 19:16:06.517779 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:06.517823 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260620 19:16:06.947285 4166 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:06.947432 4166 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:16:06.947585 4166 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
W20260620 19:16:06.947971 3754 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
W20260620 19:16:06.948055 3752 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
I20260620 19:16:06.948098 3752 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002; no voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68
I20260620 19:16:06.948173 4166 raft_consensus.cc:2726] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:16:07.304728 4167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:07.304490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:45191)
0620 19:16:07.304648 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:07.304682 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260620 19:16:07.368257 4168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:07.368021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:48425)
0620 19:16:07.368177 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:07.368215 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260620 19:16:07.518644 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:07.518331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:58003)
0620 19:16:07.518547 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:07.518598 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
I20260620 19:16:08.305363 4170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:08.305151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:36433)
0620 19:16:08.305282 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:08.305318 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
I20260620 19:16:08.368940 4171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:08.368682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:46163)
0620 19:16:08.368818 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:08.368883 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260620 19:16:08.519294 4172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:08.519095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:55091)
0620 19:16:08.519213 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:08.519249 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 19:16:09.305925 4173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:09.305739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:40665)
0620 19:16:09.305846 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:09.305880 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260620 19:16:09.369513 4174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:09.369318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:53047)
0620 19:16:09.369434 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:09.369470 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260620 19:16:09.519968 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:09.519749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:52777)
0620 19:16:09.519881 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:09.519922 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260620 19:16:10.306622 4176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:10.306372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:60377)
0620 19:16:10.306534 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:10.306577 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":79,"threads_started":1}
I20260620 19:16:10.370097 4177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:10.369901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:60505)
0620 19:16:10.370019 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:10.370053 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":69,"threads_started":1}
I20260620 19:16:10.520629 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:10.520418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:48639)
0620 19:16:10.520536 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:10.520581 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":55,"threads_started":1}
I20260620 19:16:10.939113 4179 raft_consensus.cc:482] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:10.939208 4179 raft_consensus.cc:504] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59822ff37a34691b98a99ea1114ef68" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33227 } } peers { permanent_uuid: "286bf1a9b21b492784c4fcbdbed29002" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 37803 } } peers { permanent_uuid: "881c355127524514b175bb4a19aa8693" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 41829 } }
I20260620 19:16:10.939376 4179 leader_election.cc:288] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227), 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829)
W20260620 19:16:10.939733 3752 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 881c355127524514b175bb4a19aa8693 (127.19.66.194:41829): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
W20260620 19:16:10.939816 3754 leader_election.cc:341] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c59822ff37a34691b98a99ea1114ef68 (127.19.66.195:33227): Not found: Tablet not found: d98c609dc52a42cdad8964c3a486f065
I20260620 19:16:10.939883 3754 leader_election.cc:302] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286bf1a9b21b492784c4fcbdbed29002; no voters: 881c355127524514b175bb4a19aa8693, c59822ff37a34691b98a99ea1114ef68
I20260620 19:16:10.939965 4179 raft_consensus.cc:2726] T d98c609dc52a42cdad8964c3a486f065 P 286bf1a9b21b492784c4fcbdbed29002 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 19:16:11.307310 4180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:11.307080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:44287)
0620 19:16:11.307221 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:11.307263 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260620 19:16:11.370745 4181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:11.370540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:51107)
0620 19:16:11.370661 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:11.370701 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
I20260620 19:16:11.521209 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:11.521020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:41411)
0620 19:16:11.521135 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:11.521169 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":55,"threads_started":1}
I20260620 19:16:12.308040 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:12.307806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:52541)
0620 19:16:12.307944 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:12.307993 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260620 19:16:12.371415 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:12.371186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:45639)
0620 19:16:12.371332 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:12.371372 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260620 19:16:12.521833 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:12.521615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:52981)
0620 19:16:12.521747 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:12.521789 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260620 19:16:13.308815 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:13.308583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.193:42577)
0620 19:16:13.308731 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:13.308771 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
I20260620 19:16:13.372102 4187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:13.371897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.194:42781)
0620 19:16:13.372025 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:13.372059 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260620 19:16:13.515064 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3702
I20260620 19:16:13.519496 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 3870
I20260620 19:16:13.522517 4188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:13.522334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40905 (local address 127.19.66.195:47037)
0620 19:16:13.522444 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:13.522478 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.254:40905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260620 19:16:13.523252 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4003
2026-06-20T19:16:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58013 ms)
[----------] 30 tests from SecurityITest (221713 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:13 dist-test-slave-nq8r krb5kdc[4191](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:13 dist-test-slave-nq8r krb5kdc[4191](info): set up 2 sockets
Jun 20 19:16:13 dist-test-slave-nq8r krb5kdc[4191](info): commencing operation
krb5kdc: starting...
W20260620 19:16:14.472337 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:14 dist-test-slave-nq8r krb5kdc[4191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:14.583388 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35345
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:14.662683 4204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:14.662876 4204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:14.662897 4204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:14.663967 4204 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:14.664000 4204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:14.664011 4204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:14.664022 4204 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:14.664033 4204 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:14.665426 4204 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:14.665627 4204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:14.665795 4204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:14.670306 4210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:14.670297 4209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:14.670315 4204 server_base.cc:1034] running on GCE node
W20260620 19:16:14.670298 4212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:14.670739 4204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:14.670979 4204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:14.672113 4204 hybrid_clock.cc:648] HybridClock initialized: now 1781982974672103 us; error 27 us; skew 500 ppm
Jun 20 19:16:14 dist-test-slave-nq8r krb5kdc[4191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982974, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:14.674422 4204 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:14.674829 4204 webserver.cc:460] Webserver started at http://127.19.66.254:46175/ using document root <none> and password file <none>
I20260620 19:16:14.675028 4204 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:14.675069 4204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:14.675187 4204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:14.676174 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "77bc3d8bfb754682b3d670d9bd10d0e2"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "2fc47b7bcfc279def04ac21fc0c34ff3"
server_key_iv: "94da12aa8741d9a86bb6721801d82895"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.676479 4204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "77bc3d8bfb754682b3d670d9bd10d0e2"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "2fc47b7bcfc279def04ac21fc0c34ff3"
server_key_iv: "94da12aa8741d9a86bb6721801d82895"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.677693 4204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:14.678371 4219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:14.678541 4204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:14.678632 4204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "77bc3d8bfb754682b3d670d9bd10d0e2"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "2fc47b7bcfc279def04ac21fc0c34ff3"
server_key_iv: "94da12aa8741d9a86bb6721801d82895"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.678689 4204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:14.689342 4204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:14.691340 4204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:14.691443 4204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:14.694957 4204 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:35345
I20260620 19:16:14.694988 4271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:35345 every 8 connection(s)
I20260620 19:16:14.695295 4204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:14.696097 4272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:14.697680 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4204
I20260620 19:16:14.697753 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:14.697923 19723 external_mini_cluster.cc:1428] Setting key 05ee5151e5e853f4da60e835eae965d9
I20260620 19:16:14.698472 4272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: Bootstrap starting.
I20260620 19:16:14.699025 4272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:14.699321 4272 log.cc:824] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:14.699987 4272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: No bootstrap required, opened a new log
I20260620 19:16:14.702060 4272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } }
I20260620 19:16:14.702196 4272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:14.702219 4272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bc3d8bfb754682b3d670d9bd10d0e2, State: Initialized, Role: FOLLOWER
I20260620 19:16:14.702328 4272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } }
I20260620 19:16:14.702401 4272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:14.702468 4272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:14.702523 4272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:14.703150 4272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } }
I20260620 19:16:14.703261 4272 leader_election.cc:302] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77bc3d8bfb754682b3d670d9bd10d0e2; no voters:
I20260620 19:16:14.703429 4272 leader_election.cc:288] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:14.703459 4277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:14.703611 4277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [term 1 LEADER]: Becoming Leader. State: Replica: 77bc3d8bfb754682b3d670d9bd10d0e2, State: Running, Role: LEADER
I20260620 19:16:14.703701 4272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:14.703733 4277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } }
I20260620 19:16:14.704077 4277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77bc3d8bfb754682b3d670d9bd10d0e2. Latest consensus state: current_term: 1 leader_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } } }
I20260620 19:16:14.704294 4277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:16:14 dist-test-slave-nq8r krb5kdc[4191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:14.704236 4279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bc3d8bfb754682b3d670d9bd10d0e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 35345 } } }
I20260620 19:16:14.704612 4279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2 [sys.catalog]: This master's current role is: LEADER
W20260620 19:16:14.705699 4293 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:14.705752 4293 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:14.705804 4286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:14.705946 4286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:14.707712 4286 catalog_manager.cc:1347] Generated new cluster ID: c96e2b335019413bbc7bf865b9178b56
I20260620 19:16:14.707760 4286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:14.708552 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:14.698376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.19.66.254:35345)
0620 19:16:14.698527 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:14.698534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:14.698564 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:16:14.698988 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:14.698994 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:14.699038 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:14.699162 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:14.699843 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.700781 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.701891 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.702317 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.702414 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:14.704997 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:14.705001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:14.705002 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:14.705013 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:14.707019 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:14.707387 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:14.707390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:14.707390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:14.707488 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:14.707673 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:14.707674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:14.707674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:14.707954 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:14.708093 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:14.708230 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:16:14.708354 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260620 19:16:14.736531 4286 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:14.737128 4286 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:14.760545 4286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: Generated new TSK 0
I20260620 19:16:14.760716 4286 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:16:14.789897 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:14.871223 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:14.871433 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:14.871467 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:14.872632 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:14.872674 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:14.872709 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:14.874185 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:14.874421 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:14.874631 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:14.879349 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:14.879361 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:14.879343 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:14.879602 4298 server_base.cc:1034] running on GCE node
I20260620 19:16:14.879794 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:14.880014 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:14.881156 4298 hybrid_clock.cc:648] HybridClock initialized: now 1781982974881143 us; error 33 us; skew 500 ppm
Jun 20 19:16:14 dist-test-slave-nq8r krb5kdc[4191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982974, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:14.883481 4298 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:16:14.883885 4298 webserver.cc:460] Webserver started at http://127.19.66.193:33191/ using document root <none> and password file <none>
I20260620 19:16:14.884080 4298 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:14.884127 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:14.884248 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:14.885272 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "8960bf9c2550428fae9f12c1f31aa96d"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "0507aaa601e16deacf36b84a0c8085b4"
server_key_iv: "eb79c41c5d355df42fa46706e0836bfc"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.885586 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "8960bf9c2550428fae9f12c1f31aa96d"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "0507aaa601e16deacf36b84a0c8085b4"
server_key_iv: "eb79c41c5d355df42fa46706e0836bfc"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.886756 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:14.887408 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:14.887552 4298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:14.887647 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "8960bf9c2550428fae9f12c1f31aa96d"
format_stamp: "Formatted at 2026-06-20 19:16:14 on dist-test-slave-nq8r"
server_key: "0507aaa601e16deacf36b84a0c8085b4"
server_key_iv: "eb79c41c5d355df42fa46706e0836bfc"
server_key_version: "encryptionkey@0"
I20260620 19:16:14.887705 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:14.918833 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:14.920641 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:14.920774 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:14.921006 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:14.921312 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:14.921344 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:14.921373 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:14.921391 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:14.926775 4298 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:46755
I20260620 19:16:14.926826 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:46755 every 8 connection(s)
I20260620 19:16:14.927119 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:16:14 dist-test-slave-nq8r krb5kdc[4191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982974, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:14.935073 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4298
I20260620 19:16:14.935184 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:14.935285 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:14.927786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:54377 (local address 127.19.66.254:35345)
0620 19:16:14.927848 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:14.927851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:14.928050 (+ 199us) server_negotiation.cc:407] Connection header received
0620 19:16:14.928585 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:14.928586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:14.928596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:14.928648 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:14.929564 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.930053 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.930976 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.931168 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.931211 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:14.933138 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:14.933139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:14.933139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:14.933145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:14.934181 (+ 1036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:14.934496 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:14.934497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:14.934497 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:14.934541 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:14.934722 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:14.934722 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:14.934722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:14.934809 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:14.934852 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:14.935105 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 19:16:14.935190 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 19:16:14.935400 19723 external_mini_cluster.cc:1428] Setting key 2f2d808c2bcb47c0e51c926026aaaf9e
I20260620 19:16:14.935469 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:14.927764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35345 (local address 127.19.66.193:54377)
0620 19:16:14.927917 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:14.927964 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:14.928502 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:14.928716 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:14.928719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:14.929305 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:14.929495 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:14.929500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.930116 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.930116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:14.930915 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:14.930916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:14.931222 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:14.931222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:14.931323 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:14.931791 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:14.931800 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:14.933059 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:14.934241 (+ 1182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:14.934246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:14.934248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:14.934444 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:14.934582 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:14.934583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:14.934583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:14.934681 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:14.934848 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:14.934851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:14.934986 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 19:16:14.935098 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:16:14.935255 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 19:16:14.935921 4427 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35345
I20260620 19:16:14.936017 4427 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:14.936183 4427 heartbeater.cc:511] Master 127.19.66.254:35345 requested a full tablet report, sending...
I20260620 19:16:14.936707 4236 ts_manager.cc:194] Registered new tserver with Master: 8960bf9c2550428fae9f12c1f31aa96d (127.19.66.193:46755)
I20260620 19:16:14.937420 4236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:54377
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:16:14.955093 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:15.034981 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.035189 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.035211 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.036345 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.036388 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.036413 4432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:15.037880 4432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.038101 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.038343 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:16:15.043252 4432 server_base.cc:1034] running on GCE node
W20260620 19:16:15.043252 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.043248 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.043381 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.043571 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.043823 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.044962 4432 hybrid_clock.cc:648] HybridClock initialized: now 1781982975044951 us; error 28 us; skew 500 ppm
Jun 20 19:16:15 dist-test-slave-nq8r krb5kdc[4191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982975, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:15.047250 4432 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:16:15.047654 4432 webserver.cc:460] Webserver started at http://127.19.66.194:41217/ using document root <none> and password file <none>
I20260620 19:16:15.047853 4432 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.047896 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.048012 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.049079 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "9800be37fc1e4dddbea02edcce4fa840"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "9eea5b087c85df1e587beef75959b3fb"
server_key_iv: "5cf712e8dc07c1856badb7fd6de0c967"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.049398 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "9800be37fc1e4dddbea02edcce4fa840"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "9eea5b087c85df1e587beef75959b3fb"
server_key_iv: "5cf712e8dc07c1856badb7fd6de0c967"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.050547 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:16:15.051174 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.051321 4432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:15.051396 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "9800be37fc1e4dddbea02edcce4fa840"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "9eea5b087c85df1e587beef75959b3fb"
server_key_iv: "5cf712e8dc07c1856badb7fd6de0c967"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.051457 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.063225 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.064921 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.065049 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.065264 4432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:15.065553 4432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:15.065584 4432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.065606 4432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:15.065629 4432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.071130 4432 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:40747
I20260620 19:16:15.071180 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:40747 every 8 connection(s)
I20260620 19:16:15.071471 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:16:15 dist-test-slave-nq8r krb5kdc[4191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982975, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:15.079149 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4432
I20260620 19:16:15.079237 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:15.079480 19723 external_mini_cluster.cc:1428] Setting key b4c0712256aff5347251c4dd737399d1
I20260620 19:16:15.080366 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.072084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37921 (local address 127.19.66.254:35345)
0620 19:16:15.072145 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.072147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.072343 (+ 196us) server_negotiation.cc:407] Connection header received
0620 19:16:15.072796 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.072797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.072808 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.072877 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.073787 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.074275 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.075224 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.075432 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.075481 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.077517 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.077519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.077519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:15.077526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.078929 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.079356 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.079357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.079358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.079412 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.079670 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.079671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.079671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.079820 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.079875 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.080184 (+ 309us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.080278 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:16:15.080539 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.072069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35345 (local address 127.19.66.194:37921)
0620 19:16:15.072226 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.072261 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.072720 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.072950 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.072954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.073509 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:15.073713 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.073718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.074342 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.074342 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.075164 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.075166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.075494 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.075495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.075596 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.076067 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:15.076077 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:15.077423 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:15.079029 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:15.079037 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:15.079040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:15.079295 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:15.079465 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:15.079466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:15.079466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:15.079594 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:15.079880 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:15.079886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:15.080048 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.080160 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.080320 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260620 19:16:15.081048 4561 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35345
I20260620 19:16:15.081146 4561 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:15.081323 4561 heartbeater.cc:511] Master 127.19.66.254:35345 requested a full tablet report, sending...
I20260620 19:16:15.081710 4236 ts_manager.cc:194] Registered new tserver with Master: 9800be37fc1e4dddbea02edcce4fa840 (127.19.66.194:40747)
I20260620 19:16:15.082201 4236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:37921
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:16:15.100679 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:46865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:15.178426 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.178642 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.178673 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.179787 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.179824 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.179859 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:15.181303 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:46865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.181529 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.181730 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:15.186342 4571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.186367 4572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.186558 4574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.186558 4566 server_base.cc:1034] running on GCE node
I20260620 19:16:15.186784 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.187012 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.188155 4566 hybrid_clock.cc:648] HybridClock initialized: now 1781982975188145 us; error 28 us; skew 500 ppm
Jun 20 19:16:15 dist-test-slave-nq8r krb5kdc[4191](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982975, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:15.190606 4566 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:16:15.190994 4566 webserver.cc:460] Webserver started at http://127.19.66.195:43033/ using document root <none> and password file <none>
I20260620 19:16:15.191195 4566 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.191236 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.191354 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.192380 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "fe6e58f7d04b44179c1626af0c64f9d3"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "a03b611243ed4ac5d752cee24e3d9d78"
server_key_iv: "ce073246d3242d233add1f36feef7c02"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.192700 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "fe6e58f7d04b44179c1626af0c64f9d3"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "a03b611243ed4ac5d752cee24e3d9d78"
server_key_iv: "ce073246d3242d233add1f36feef7c02"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.193895 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:15.194541 4581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.194692 4566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:15.194770 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "fe6e58f7d04b44179c1626af0c64f9d3"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "a03b611243ed4ac5d752cee24e3d9d78"
server_key_iv: "ce073246d3242d233add1f36feef7c02"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.194828 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.225800 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.227944 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.228081 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.228291 4566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:15.228570 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:15.228602 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.228631 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:15.228663 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.233774 4566 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:41793
I20260620 19:16:15.233839 4694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:41793 every 8 connection(s)
I20260620 19:16:15.234115 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:16:15.234936 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4566
I20260620 19:16:15.235028 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:15.235198 19723 external_mini_cluster.cc:1428] Setting key 8a114b3869c760effd78e4c86417b752
Jun 20 19:16:15 dist-test-slave-nq8r krb5kdc[4191](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982975, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:15.242661 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.234805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:44759 (local address 127.19.66.254:35345)
0620 19:16:15.234871 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.234873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.235016 (+ 143us) server_negotiation.cc:407] Connection header received
0620 19:16:15.235555 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.235556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.235567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.235601 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.236599 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.237124 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.238129 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.238318 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.238374 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.240276 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.240278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.240279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:15.240286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.241431 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.241813 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.241814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.241815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.241868 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.242068 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.242069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.242069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.242188 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.242242 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.242491 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.242572 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:16:15.242828 4697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.234740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:35345 (local address 127.19.66.195:44759)
0620 19:16:15.234898 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.234938 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.235472 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.235685 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.235689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.236312 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:15.236503 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.236508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.237211 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.237212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.238041 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.238043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.238388 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.238389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.238528 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.238938 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:15.238945 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:15.240188 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:15.241506 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:15.241512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:15.241514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:15.241749 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:15.241920 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:15.241920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:15.241921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:15.242016 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:15.242243 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:15.242249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:15.242377 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.242465 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.242624 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260620 19:16:15.243216 4695 heartbeater.cc:347] Connected to a master server at 127.19.66.254:35345
I20260620 19:16:15.243297 4695 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:15.243433 4695 heartbeater.cc:511] Master 127.19.66.254:35345 requested a full tablet report, sending...
I20260620 19:16:15.243736 4236 ts_manager.cc:194] Registered new tserver with Master: fe6e58f7d04b44179c1626af0c64f9d3 (127.19.66.195:41793)
I20260620 19:16:15.244055 4236 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:44759
I20260620 19:16:15.246651 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:15.252767 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.247878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.19.66.254:35345)
0620 19:16:15.247928 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.247929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.248037 (+ 108us) server_negotiation.cc:407] Connection header received
0620 19:16:15.248076 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.248077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.248085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.248135 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.248598 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.249105 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.249769 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.249937 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.249974 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.250960 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.250960 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.250960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:15.250965 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.251985 (+ 1020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.252253 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.252253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.252253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.252296 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:15.252402 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:15.252402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:15.252403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:15.252518 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.252585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.252658 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.252708 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:15.254207 4236 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:15.255777 4236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:15.261632 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.19.66.195:41793)
0620 19:16:15.258026 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.258031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.258142 (+ 111us) server_negotiation.cc:407] Connection header received
0620 19:16:15.258263 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.258266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.258281 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.258343 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.258548 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259251 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.260604 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261211 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261247 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261289 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.261512 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.261553 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":69,"threads_started":1}
I20260620 19:16:15.261638 4705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:41793 (local address 127.0.0.1:45522)
0620 19:16:15.258095 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.258102 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.258165 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.258362 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.258363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.258364 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.258394 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.258395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259342 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.259343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.260513 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.260515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261340 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.261401 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261416 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.261486 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.261546 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":8,"thread_start_us":59,"threads_started":1}
I20260620 19:16:15.261823 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46755 (local address 127.0.0.1:46922)
0620 19:16:15.257973 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.257983 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.258033 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.258326 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.258327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.258329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.258397 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.258398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259629 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.259630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.260701 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.260702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261597 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.261669 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261673 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.261721 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.261758 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260620 19:16:15.261895 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.19.66.194:40747)
0620 19:16:15.257997 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.258001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.258009 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:15.258038 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.258039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.258052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.258122 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.258413 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259563 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.260561 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261240 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261311 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261362 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.261780 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.261822 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260620 19:16:15.262022 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.19.66.193:46755)
0620 19:16:15.258107 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.258112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.258121 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:16:15.258174 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.258176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.258190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.258258 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.258463 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259527 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.260774 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261459 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261518 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261565 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.261914 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.261952 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":89,"threads_started":1}
I20260620 19:16:15.262420 4629 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a (DEFAULT_TABLE table=test-table [id=074ae526c7ae4f36b48d5cff1d037d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.262732 4629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab2d8bb322e4c94b4c1365414914a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.262615 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a (DEFAULT_TABLE table=test-table [id=074ae526c7ae4f36b48d5cff1d037d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.262948 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab2d8bb322e4c94b4c1365414914a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.265105 4709 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Bootstrap starting.
I20260620 19:16:15.265250 4710 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: Bootstrap starting.
I20260620 19:16:15.261637 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.257596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:40747 (local address 127.0.0.1:57770)
0620 19:16:15.257759 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.257794 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.257857 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.258127 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.258130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.258132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.258192 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.258194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.259658 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.259659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.260481 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.260484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.261320 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.261321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.261401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.261416 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.261484 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.261546 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":25,"thread_start_us":69,"threads_started":1}
I20260620 19:16:15.265798 4709 tablet_bootstrap.cc:654] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.265798 4710 tablet_bootstrap.cc:654] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.266013 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a (DEFAULT_TABLE table=test-table [id=074ae526c7ae4f36b48d5cff1d037d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.266309 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab2d8bb322e4c94b4c1365414914a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.266373 4709 log.cc:824] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.266372 4710 log.cc:824] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.267056 4710 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: No bootstrap required, opened a new log
I20260620 19:16:15.267132 4710 ts_tablet_manager.cc:1397] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:15.267290 4709 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: No bootstrap required, opened a new log
I20260620 19:16:15.267407 4709 ts_tablet_manager.cc:1397] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:15.268838 4713 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Bootstrap starting.
I20260620 19:16:15.269064 4710 raft_consensus.cc:348] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.269265 4710 raft_consensus.cc:374] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.269301 4710 raft_consensus.cc:729] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8960bf9c2550428fae9f12c1f31aa96d, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.269204 4709 raft_consensus.cc:348] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.269342 4709 raft_consensus.cc:374] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.269393 4709 raft_consensus.cc:729] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe6e58f7d04b44179c1626af0c64f9d3, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.269475 4710 consensus_queue.cc:260] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.269493 4709 consensus_queue.cc:260] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.269673 4713 tablet_bootstrap.cc:654] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.269762 4709 ts_tablet_manager.cc:1428] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:16:15.269762 4710 ts_tablet_manager.cc:1428] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 19:16:15.269845 4695 heartbeater.cc:503] Master 127.19.66.254:35345 was elected leader, sending a full tablet report...
I20260620 19:16:15.270186 4427 heartbeater.cc:503] Master 127.19.66.254:35345 was elected leader, sending a full tablet report...
I20260620 19:16:15.270220 4713 log.cc:824] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.270991 4713 tablet_bootstrap.cc:492] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: No bootstrap required, opened a new log
I20260620 19:16:15.271059 4713 ts_tablet_manager.cc:1397] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:16:15.272315 4713 raft_consensus.cc:348] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.272440 4713 raft_consensus.cc:374] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.272460 4713 raft_consensus.cc:729] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9800be37fc1e4dddbea02edcce4fa840, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.272543 4713 consensus_queue.cc:260] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.272758 4713 ts_tablet_manager.cc:1428] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:16:15.272794 4561 heartbeater.cc:503] Master 127.19.66.254:35345 was elected leader, sending a full tablet report...
W20260620 19:16:15.322036 4562 tablet.cc:2369] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:16:15.346935 4717 raft_consensus.cc:482] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:15.347055 4717 raft_consensus.cc:504] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.347419 4717 leader_election.cc:288] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8960bf9c2550428fae9f12c1f31aa96d (127.19.66.193:46755), fe6e58f7d04b44179c1626af0c64f9d3 (127.19.66.195:41793)
I20260620 19:16:15.350310 4707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.347654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37023 (local address 127.19.66.195:41793)
0620 19:16:15.347707 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.347709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.347915 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:16:15.347979 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.347980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.347989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.348042 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.348196 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.348730 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.349644 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.350018 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.350079 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.350086 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.350212 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.350247 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:16:15.350342 4706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.347739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:41583 (local address 127.19.66.193:46755)
0620 19:16:15.347793 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.347795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.347803 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:15.347829 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.347831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.347843 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.347908 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.348072 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.348665 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.349645 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.350014 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.350076 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.350085 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.350236 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.350272 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:16:15.350356 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.347578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46755 (local address 127.19.66.194:41583)
0620 19:16:15.347643 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.347656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.347721 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.347896 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.347897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.347899 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.347982 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.347984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.348743 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.348744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.349545 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.349548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.350091 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.350092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.350177 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.350179 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.350222 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.350262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":24}
I20260620 19:16:15.350365 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.347578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:41793 (local address 127.19.66.194:37023)
0620 19:16:15.347855 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.347866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.347927 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.348057 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.348057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.348059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.348136 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.348138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.348803 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.348804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.349559 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.349560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.350089 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.350090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.350150 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.350155 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.350199 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.350239 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":47,"thread_start_us":145,"threads_started":1}
I20260620 19:16:15.350682 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab2d8bb322e4c94b4c1365414914a2a" candidate_uuid: "9800be37fc1e4dddbea02edcce4fa840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8960bf9c2550428fae9f12c1f31aa96d" is_pre_election: true
I20260620 19:16:15.350836 4381 raft_consensus.cc:2454] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800be37fc1e4dddbea02edcce4fa840 in term 0.
I20260620 19:16:15.350816 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab2d8bb322e4c94b4c1365414914a2a" candidate_uuid: "9800be37fc1e4dddbea02edcce4fa840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" is_pre_election: true
I20260620 19:16:15.350953 4649 raft_consensus.cc:2454] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9800be37fc1e4dddbea02edcce4fa840 in term 0.
I20260620 19:16:15.351052 4451 leader_election.cc:302] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8960bf9c2550428fae9f12c1f31aa96d, 9800be37fc1e4dddbea02edcce4fa840; no voters:
I20260620 19:16:15.351174 4717 raft_consensus.cc:2781] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:15.351230 4717 raft_consensus.cc:482] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:15.351253 4717 raft_consensus.cc:3037] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.351918 4717 raft_consensus.cc:504] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.352027 4717 leader_election.cc:288] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [CANDIDATE]: Term 1 election: Requested vote from peers 8960bf9c2550428fae9f12c1f31aa96d (127.19.66.193:46755), fe6e58f7d04b44179c1626af0c64f9d3 (127.19.66.195:41793)
I20260620 19:16:15.352173 4649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab2d8bb322e4c94b4c1365414914a2a" candidate_uuid: "9800be37fc1e4dddbea02edcce4fa840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe6e58f7d04b44179c1626af0c64f9d3"
I20260620 19:16:15.352188 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab2d8bb322e4c94b4c1365414914a2a" candidate_uuid: "9800be37fc1e4dddbea02edcce4fa840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8960bf9c2550428fae9f12c1f31aa96d"
I20260620 19:16:15.352247 4649 raft_consensus.cc:3037] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.352252 4381 raft_consensus.cc:3037] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.352941 4649 raft_consensus.cc:2454] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800be37fc1e4dddbea02edcce4fa840 in term 1.
I20260620 19:16:15.352983 4381 raft_consensus.cc:2454] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9800be37fc1e4dddbea02edcce4fa840 in term 1.
I20260620 19:16:15.353124 4451 leader_election.cc:302] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8960bf9c2550428fae9f12c1f31aa96d, 9800be37fc1e4dddbea02edcce4fa840; no voters:
I20260620 19:16:15.353219 4717 raft_consensus.cc:2781] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:15.353370 4717 raft_consensus.cc:686] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 1 LEADER]: Becoming Leader. State: Replica: 9800be37fc1e4dddbea02edcce4fa840, State: Running, Role: LEADER
I20260620 19:16:15.353454 4717 consensus_queue.cc:237] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } }
I20260620 19:16:15.354219 4235 catalog_manager.cc:5495] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9800be37fc1e4dddbea02edcce4fa840 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "9800be37fc1e4dddbea02edcce4fa840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9800be37fc1e4dddbea02edcce4fa840" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 40747 } health_report { overall_health: HEALTHY } } }
I20260620 19:16:15.375386 4708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.373230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57782 (local address 127.19.66.194:40747)
0620 19:16:15.373292 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.373294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.373302 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:15.373337 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.373338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.373361 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.373419 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:15.373536 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.373992 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.374705 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.374889 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.374940 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.375079 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:15.375182 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:15.375249 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.375275 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.375296 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:16:15.377025 4381 raft_consensus.cc:1261] T 5ab2d8bb322e4c94b4c1365414914a2a P 8960bf9c2550428fae9f12c1f31aa96d [term 1 FOLLOWER]: Refusing update from remote peer 9800be37fc1e4dddbea02edcce4fa840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:15.377029 4649 raft_consensus.cc:1261] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 1 FOLLOWER]: Refusing update from remote peer 9800be37fc1e4dddbea02edcce4fa840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:15.377319 4719 consensus_queue.cc:1035] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe6e58f7d04b44179c1626af0c64f9d3" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:15.377434 4717 consensus_queue.cc:1035] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8960bf9c2550428fae9f12c1f31aa96d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 46755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:15.380312 4724 mvcc.cc:205] Tried to move back new op lower bound from 7299002267141808128 to 7299002267048267776. Current Snapshot: MvccSnapshot[applied={T|T < 7299002267141808128}]
I20260620 19:16:15.381035 4723 mvcc.cc:205] Tried to move back new op lower bound from 7299002267141808128 to 7299002267048267776. Current Snapshot: MvccSnapshot[applied={T|T < 7299002267141808128 or (T in {7299002267141808128})}]
I20260620 19:16:15.381871 4235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:15.381960 4235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:15.382675 4235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77bc3d8bfb754682b3d670d9bd10d0e2: Sending DeleteTablet for 3 replicas of tablet 5ab2d8bb322e4c94b4c1365414914a2a
I20260620 19:16:15.383008 4361 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:15 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:46922
I20260620 19:16:15.383020 4629 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:15 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:45522
I20260620 19:16:15.383217 4495 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab2d8bb322e4c94b4c1365414914a2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:15 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:57770
I20260620 19:16:15.383405 4733 tablet_replica.cc:330] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: stopping tablet replica
I20260620 19:16:15.383538 4733 raft_consensus.cc:2229] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:16:15.383634 4733 raft_consensus.cc:2258] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:15.383773 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4298
I20260620 19:16:15.384032 4733 ts_tablet_manager.cc:1905] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:15.386483 4733 ts_tablet_manager.cc:1918] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:15.386629 4733 log.cc:1196] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/5ab2d8bb322e4c94b4c1365414914a2a
I20260620 19:16:15.386945 4733 ts_tablet_manager.cc:1939] T 5ab2d8bb322e4c94b4c1365414914a2a P 9800be37fc1e4dddbea02edcce4fa840: Deleting consensus metadata
I20260620 19:16:15.387306 4222 catalog_manager.cc:4841] TS 9800be37fc1e4dddbea02edcce4fa840 (127.19.66.194:40747): tablet 5ab2d8bb322e4c94b4c1365414914a2a (table test-table [id=074ae526c7ae4f36b48d5cff1d037d1d]) successfully deleted
W20260620 19:16:15.389818 4223 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:46755 (error 108)
I20260620 19:16:15.389889 4731 tablet_replica.cc:330] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: stopping tablet replica
I20260620 19:16:15.389997 4731 raft_consensus.cc:2229] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:15.390110 4731 raft_consensus.cc:2258] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:15.390290 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4432
I20260620 19:16:15.390794 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.390273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:46755 (local address 127.0.0.1:46926)
0620 19:16:15.390711 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.390755 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
W20260620 19:16:15.390935 4223 catalog_manager.cc:4584] TS 8960bf9c2550428fae9f12c1f31aa96d (127.19.66.193:46755): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ab2d8bb322e4c94b4c1365414914a2a: Network error: Client connection negotiation failed: client connection to 127.19.66.193:46755: connect: Connection refused (error 111)
I20260620 19:16:15.391886 4731 ts_tablet_manager.cc:1905] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:15.393393 4731 ts_tablet_manager.cc:1918] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:15.393465 4731 log.cc:1196] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/5ab2d8bb322e4c94b4c1365414914a2a
I20260620 19:16:15.393716 4731 ts_tablet_manager.cc:1939] T 5ab2d8bb322e4c94b4c1365414914a2a P fe6e58f7d04b44179c1626af0c64f9d3: Deleting consensus metadata
I20260620 19:16:15.394097 4221 catalog_manager.cc:4841] TS fe6e58f7d04b44179c1626af0c64f9d3 (127.19.66.195:41793): tablet 5ab2d8bb322e4c94b4c1365414914a2a (table test-table [id=074ae526c7ae4f36b48d5cff1d037d1d]) successfully deleted
I20260620 19:16:15.395540 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4566
I20260620 19:16:15.399806 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4204
2026-06-20T19:16:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1876 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T19:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:15Z Disabled control of system clock
I20260620 19:16:15.424544 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37991
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:37991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:15.502732 4738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.502939 4738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.502959 4738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.504066 4738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:15.504101 4738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.504112 4738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.504122 4738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:15.504132 4738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:15.505476 4738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:37991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:37991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.505684 4738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.505946 4738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:15.510450 4746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.510509 4743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.510459 4744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.510579 4738 server_base.cc:1034] running on GCE node
I20260620 19:16:15.510955 4738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.511240 4738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.512393 4738 hybrid_clock.cc:648] HybridClock initialized: now 1781982975512371 us; error 33 us; skew 500 ppm
I20260620 19:16:15.513608 4738 webserver.cc:460] Webserver started at http://127.19.66.254:42701/ using document root <none> and password file <none>
I20260620 19:16:15.513825 4738 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.513875 4738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.513998 4738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.515033 4738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "9c14e2793cca4ac69574deb4d65430e2"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b9084c8a67e8054b145a75066e06741e"
server_key_iv: "2ec8c0af88fb7b0d94c241031d5cf36c"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.515370 4738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "9c14e2793cca4ac69574deb4d65430e2"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b9084c8a67e8054b145a75066e06741e"
server_key_iv: "2ec8c0af88fb7b0d94c241031d5cf36c"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.516516 4738 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:15.517212 4752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.517360 4738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.517449 4738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "9c14e2793cca4ac69574deb4d65430e2"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b9084c8a67e8054b145a75066e06741e"
server_key_iv: "2ec8c0af88fb7b0d94c241031d5cf36c"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.517505 4738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.538650 4738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.538930 4738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.539032 4738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.542701 4738 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:37991
I20260620 19:16:15.542735 4804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:37991 every 8 connection(s)
I20260620 19:16:15.543072 4738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:15.544003 4805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.546289 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: Bootstrap starting.
I20260620 19:16:15.546860 4805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.547122 4805 log.cc:824] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.547755 4805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: No bootstrap required, opened a new log
I20260620 19:16:15.548277 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4738
I20260620 19:16:15.548367 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:15.548588 19723 external_mini_cluster.cc:1428] Setting key 932266a04dc22f613e705f2c442c5e34
I20260620 19:16:15.549566 4805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } }
I20260620 19:16:15.549724 4805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.549759 4805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c14e2793cca4ac69574deb4d65430e2, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.549888 4805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } }
I20260620 19:16:15.549968 4805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:15.549993 4805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:15.550040 4805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.550531 4805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } }
I20260620 19:16:15.550621 4805 leader_election.cc:302] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c14e2793cca4ac69574deb4d65430e2; no voters:
I20260620 19:16:15.550766 4805 leader_election.cc:288] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:15.550805 4810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:15.550946 4810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9c14e2793cca4ac69574deb4d65430e2, State: Running, Role: LEADER
I20260620 19:16:15.551041 4805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:15.551072 4810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } }
I20260620 19:16:15.551488 4810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c14e2793cca4ac69574deb4d65430e2. Latest consensus state: current_term: 1 leader_uuid: "9c14e2793cca4ac69574deb4d65430e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } } }
I20260620 19:16:15.551582 4810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:15.551702 4812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c14e2793cca4ac69574deb4d65430e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c14e2793cca4ac69574deb4d65430e2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 37991 } } }
I20260620 19:16:15.551784 4816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:15.552294 4816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:15.551784 4812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:15.553836 4816 catalog_manager.cc:1347] Generated new cluster ID: 9b684579c92f44b4a45617cc4ed03f27
I20260620 19:16:15.553884 4816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:15.556044 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.549090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.19.66.254:37991)
0620 19:16:15.549272 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.549279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.549304 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:16:15.549523 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.549529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.549585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.549700 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.549938 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.552779 (+ 2841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.553907 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.554481 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.554596 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.555316 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.555320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.555320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:15.555523 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.555578 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:15.555596 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.555665 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.555732 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.555819 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260620 19:16:15.560577 4816 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:15.561228 4816 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:15.576053 4816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: Generated new TSK 0
I20260620 19:16:15.576267 4816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:15.580538 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--builtin_ntp_servers=127.19.66.212:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:15.662119 4829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.662335 4829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.662366 4829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.663538 4829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.663582 4829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.663617 4829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:15.665035 4829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.665274 4829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.665560 4829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:15.670226 4837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.670243 4835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.670238 4834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.670740 4829 server_base.cc:1034] running on GCE node
I20260620 19:16:15.670912 4829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.671135 4829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.672284 4829 hybrid_clock.cc:648] HybridClock initialized: now 1781982975672272 us; error 30 us; skew 500 ppm
I20260620 19:16:15.673448 4829 webserver.cc:460] Webserver started at http://127.19.66.193:42495/ using document root <none> and password file <none>
I20260620 19:16:15.673668 4829 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.673717 4829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.673844 4829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.674870 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "351fd9e18fb746a38754e46f8e7b70a5"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b3579f83aa615dda99cf74b477cff74d"
server_key_iv: "3aac488e0aae88c165ff29e381dad92f"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.675200 4829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "351fd9e18fb746a38754e46f8e7b70a5"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b3579f83aa615dda99cf74b477cff74d"
server_key_iv: "3aac488e0aae88c165ff29e381dad92f"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.676352 4829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:16:15.677031 4843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.677193 4829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.677274 4829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "351fd9e18fb746a38754e46f8e7b70a5"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "b3579f83aa615dda99cf74b477cff74d"
server_key_iv: "3aac488e0aae88c165ff29e381dad92f"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.677340 4829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.697228 4829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.697489 4829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.697607 4829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.697796 4829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:15.698113 4829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:15.698146 4829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.698174 4829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:15.698199 4829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.703478 4829 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:41677
I20260620 19:16:15.703539 4956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:41677 every 8 connection(s)
I20260620 19:16:15.703805 4829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:16:15.705519 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4829
I20260620 19:16:15.705615 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:15.705822 19723 external_mini_cluster.cc:1428] Setting key 997db5a9804b77f0b3e55e9e5de5dd67
I20260620 19:16:15.706698 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--builtin_ntp_servers=127.19.66.212:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:16:15.709367 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.704398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:35393 (local address 127.19.66.254:37991)
0620 19:16:15.704448 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.704450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.704676 (+ 226us) server_negotiation.cc:407] Connection header received
0620 19:16:15.705205 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.705207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.705221 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.705271 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.706099 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.706700 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.708020 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.708305 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.708373 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.709119 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.709121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.709121 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:15.709129 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.709145 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:15.709149 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.709206 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.709262 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.709289 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:16:15.709506 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.704381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37991 (local address 127.19.66.193:35393)
0620 19:16:15.704552 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.704590 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.705113 (+ 523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.705338 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.705343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.705353 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:15.706017 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.706024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.706793 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.706794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.707914 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.707918 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.708380 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.708381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.708484 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.709000 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:15.709000 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:15.709027 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:15.709048 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:15.709207 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:15.709209 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:15.709211 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.709254 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.709327 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260620 19:16:15.709956 4957 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37991
I20260620 19:16:15.710040 4957 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:15.710203 4957 heartbeater.cc:511] Master 127.19.66.254:37991 requested a full tablet report, sending...
I20260620 19:16:15.710700 4769 ts_manager.cc:194] Registered new tserver with Master: 351fd9e18fb746a38754e46f8e7b70a5 (127.19.66.193:41677)
I20260620 19:16:15.711586 4769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.193:35393
W20260620 19:16:15.784801 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.785014 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.785034 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.786109 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.786140 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.786162 4960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:15.787420 4960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.787607 4960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.787855 4960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:15.792423 4968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.792447 4966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.792475 4965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.792726 4960 server_base.cc:1034] running on GCE node
I20260620 19:16:15.792937 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.793130 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.794287 4960 hybrid_clock.cc:648] HybridClock initialized: now 1781982975794272 us; error 28 us; skew 500 ppm
I20260620 19:16:15.795434 4960 webserver.cc:460] Webserver started at http://127.19.66.194:39155/ using document root <none> and password file <none>
I20260620 19:16:15.795636 4960 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.795699 4960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.795827 4960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.796898 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "9f960c2a35e241928cda8db7ac40d01b"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "bdc18e33e3c216a2692900f19a488916"
server_key_iv: "2dce1912d4cd5c12e8467f6d74510453"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.797231 4960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "9f960c2a35e241928cda8db7ac40d01b"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "bdc18e33e3c216a2692900f19a488916"
server_key_iv: "2dce1912d4cd5c12e8467f6d74510453"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.798367 4960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:16:15.799005 4974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.799129 4960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:15.799204 4960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "9f960c2a35e241928cda8db7ac40d01b"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "bdc18e33e3c216a2692900f19a488916"
server_key_iv: "2dce1912d4cd5c12e8467f6d74510453"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.799270 4960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.815999 4960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.816321 4960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.816449 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.816690 4960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:15.817061 4960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:15.817095 4960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.817117 4960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:15.817139 4960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.822619 4960 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:46775
I20260620 19:16:15.822675 5087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:46775 every 8 connection(s)
I20260620 19:16:15.822978 4960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:16:15.827754 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.823651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:55401 (local address 127.19.66.254:37991)
0620 19:16:15.823720 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.823722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.823912 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:16:15.824374 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.824375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.824384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.824443 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.825131 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.825620 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.826583 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.826773 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.826826 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.827550 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.827551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.827552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:15.827556 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.827568 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:15.827571 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.827610 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.827672 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.827695 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 19:16:15.827944 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.823625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37991 (local address 127.19.66.194:55401)
0620 19:16:15.823795 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.823835 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.824299 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.824506 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.824509 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.824517 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:15.825037 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.825041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.825696 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.825697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.826477 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.826479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.826842 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.826843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.826946 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.827404 (+ 458us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:15.827404 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:15.827431 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:15.827452 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:15.827619 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:15.827623 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:15.827625 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.827669 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.827744 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260620 19:16:15.828387 5088 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37991
I20260620 19:16:15.828469 5088 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:15.828640 5088 heartbeater.cc:511] Master 127.19.66.254:37991 requested a full tablet report, sending...
I20260620 19:16:15.829015 4769 ts_manager.cc:194] Registered new tserver with Master: 9f960c2a35e241928cda8db7ac40d01b (127.19.66.194:46775)
I20260620 19:16:15.829345 4769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.194:55401
I20260620 19:16:15.832252 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 4960
I20260620 19:16:15.832329 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:15.832484 19723 external_mini_cluster.cc:1428] Setting key 97eba419c9e83c8843032adbb062a33c
I20260620 19:16:15.833297 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--builtin_ntp_servers=127.19.66.212:38547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:15.910254 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:15.910439 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:15.910458 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:15.911558 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:15.911595 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:15.911621 5091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:15.913030 5091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:37991
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:15.913254 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:15.913518 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:15.918125 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.918144 5091 server_base.cc:1034] running on GCE node
W20260620 19:16:15.918129 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:15.918131 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:15.918605 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:15.918821 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:15.919970 5091 hybrid_clock.cc:648] HybridClock initialized: now 1781982975919956 us; error 30 us; skew 500 ppm
I20260620 19:16:15.921272 5091 webserver.cc:460] Webserver started at http://127.19.66.195:39533/ using document root <none> and password file <none>
I20260620 19:16:15.921497 5091 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:15.921541 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:15.921662 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:15.922688 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "304cf9c27d2b4255a15c34d2fee87f9d"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "6f322a3896b90a7b8f293a525a9d1633"
server_key_iv: "571df7653b1e4255476077fd36851be3"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.922997 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "304cf9c27d2b4255a15c34d2fee87f9d"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "6f322a3896b90a7b8f293a525a9d1633"
server_key_iv: "571df7653b1e4255476077fd36851be3"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.924245 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:15.924995 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.925122 5091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:15.925201 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "304cf9c27d2b4255a15c34d2fee87f9d"
format_stamp: "Formatted at 2026-06-20 19:16:15 on dist-test-slave-nq8r"
server_key: "6f322a3896b90a7b8f293a525a9d1633"
server_key_iv: "571df7653b1e4255476077fd36851be3"
server_key_version: "encryptionkey@0"
I20260620 19:16:15.925268 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:15.938328 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:15.938585 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:15.938705 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:15.938886 5091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:15.939164 5091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:15.939196 5091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.939216 5091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:15.939240 5091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:15.944519 5091 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:43319
I20260620 19:16:15.944571 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:43319 every 8 connection(s)
I20260620 19:16:15.944887 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:16:15.947394 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5091
I20260620 19:16:15.947484 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:15.947686 19723 external_mini_cluster.cc:1428] Setting key 45180012bc932051a503107870b73c19
I20260620 19:16:15.949896 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.945541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:52811 (local address 127.19.66.254:37991)
0620 19:16:15.945599 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.945601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.945823 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:16:15.946305 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.946306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.946318 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.946365 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.946993 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.947515 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.948743 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.948953 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.949004 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.949726 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.949727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.949727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:15.949731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.949740 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:15.949743 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.949772 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.949823 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.949842 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:15.950080 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.945516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:37991 (local address 127.19.66.195:52811)
0620 19:16:15.945691 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.945723 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.946211 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.946442 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.946446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.946455 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:15.946911 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.946915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.947594 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.947596 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.948671 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.948674 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.949008 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.949009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.949112 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.949609 (+ 497us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:15.949610 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:15.949644 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:15.949666 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:15.949779 (+ 113us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:15.949781 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:15.949783 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.949824 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.949896 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260620 19:16:15.950460 5219 heartbeater.cc:347] Connected to a master server at 127.19.66.254:37991
I20260620 19:16:15.950546 5219 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:15.950711 5219 heartbeater.cc:511] Master 127.19.66.254:37991 requested a full tablet report, sending...
I20260620 19:16:15.951025 4769 ts_manager.cc:194] Registered new tserver with Master: 304cf9c27d2b4255a15c34d2fee87f9d (127.19.66.195:43319)
I20260620 19:16:15.951385 4769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.195:52811
I20260620 19:16:15.959283 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:15.963136 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.960495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.19.66.254:37991)
0620 19:16:15.960546 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.960548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.960644 (+ 96us) server_negotiation.cc:407] Connection header received
0620 19:16:15.960705 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.960706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.960718 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.960763 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:15.960939 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.961408 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.962097 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.962300 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.962352 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.962921 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:15.962922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:15.962922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:15.962926 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:15.962940 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:15.962943 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:15.962985 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.963035 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.963056 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 19:16:15.964505 4769 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37838:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:15.966025 4769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:15.971688 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.967988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.19.66.195:43319)
0620 19:16:15.968141 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.968146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.968353 (+ 207us) server_negotiation.cc:407] Connection header received
0620 19:16:15.968396 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.968398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.968414 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.968481 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.968622 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969435 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.970752 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971331 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971389 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.971580 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.971614 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260620 19:16:15.971698 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.967855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:43319 (local address 127.0.0.1:38704)
0620 19:16:15.968174 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.968212 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.968278 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.968494 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.968496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.968498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.968553 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.968555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969508 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.969510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.970661 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.970664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971408 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.971482 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971496 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.971558 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.971609 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":52,"threads_started":1}
I20260620 19:16:15.971884 5228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.967855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41677 (local address 127.0.0.1:58638)
0620 19:16:15.968200 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.968231 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.968286 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.968548 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.968548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.968550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.968576 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.968577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969879 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.969880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.970598 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.970601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971500 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.971560 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971564 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.971610 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.971647 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":20,"thread_start_us":61,"threads_started":1}
I20260620 19:16:15.971975 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.967927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.19.66.193:41677)
0620 19:16:15.968206 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.968212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.968356 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:16:15.968401 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.968403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.968419 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.968482 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.968642 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969788 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.970685 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971321 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971382 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971466 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.971689 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.971723 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260620 19:16:15.972066 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.968047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46775 (local address 127.0.0.1:43046)
0620 19:16:15.968323 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.968331 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.968380 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.968577 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.968578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.968580 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.968623 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.968625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969879 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.969880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.970960 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.970963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971866 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.971920 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971924 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.971969 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.972001 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":23,"thread_start_us":68,"threads_started":1}
I20260620 19:16:15.972268 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.968154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.19.66.194:46775)
0620 19:16:15.968306 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.968309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.968379 (+ 70us) server_negotiation.cc:407] Connection header received
0620 19:16:15.968442 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.968444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.968457 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.968520 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.968718 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.969799 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971032 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.971701 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.971755 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.971837 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.972155 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.972196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260620 19:16:15.972535 5153 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a306d8024475f9b234e4ec8d41418 (DEFAULT_TABLE table=test-table [id=3d358f1b4a1947e694cc506d8c25ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.972612 4891 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a306d8024475f9b234e4ec8d41418 (DEFAULT_TABLE table=test-table [id=3d358f1b4a1947e694cc506d8c25ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.972785 5022 tablet_service.cc:1467] Processing CreateTablet for tablet fd0a306d8024475f9b234e4ec8d41418 (DEFAULT_TABLE table=test-table [id=3d358f1b4a1947e694cc506d8c25ef46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:15.972923 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a306d8024475f9b234e4ec8d41418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.973147 4891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a306d8024475f9b234e4ec8d41418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.973227 5022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0a306d8024475f9b234e4ec8d41418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:15.975525 5233 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Bootstrap starting.
I20260620 19:16:15.976105 5234 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Bootstrap starting.
I20260620 19:16:15.976269 5233 tablet_bootstrap.cc:654] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.976517 5233 log.cc:824] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.976626 5234 tablet_bootstrap.cc:654] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.976946 5234 log.cc:824] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.976989 5235 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: Bootstrap starting.
I20260620 19:16:15.977303 5233 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: No bootstrap required, opened a new log
I20260620 19:16:15.977368 5233 ts_tablet_manager.cc:1397] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:15.977561 5234 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: No bootstrap required, opened a new log
I20260620 19:16:15.977581 5235 tablet_bootstrap.cc:654] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:15.977617 5234 ts_tablet_manager.cc:1397] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:16:15.977864 5235 log.cc:824] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:15.978600 5235 tablet_bootstrap.cc:492] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: No bootstrap required, opened a new log
I20260620 19:16:15.978674 5235 ts_tablet_manager.cc:1397] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:15.979657 5234 raft_consensus.cc:348] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.979784 5234 raft_consensus.cc:374] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.979815 5234 raft_consensus.cc:729] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304cf9c27d2b4255a15c34d2fee87f9d, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.979945 5234 consensus_queue.cc:260] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.979947 5233 raft_consensus.cc:348] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.980139 5233 raft_consensus.cc:374] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.980181 5233 raft_consensus.cc:729] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f960c2a35e241928cda8db7ac40d01b, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.980199 5234 ts_tablet_manager.cc:1428] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:16:15.980320 5219 heartbeater.cc:503] Master 127.19.66.254:37991 was elected leader, sending a full tablet report...
I20260620 19:16:15.980305 5233 consensus_queue.cc:260] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.980535 5233 ts_tablet_manager.cc:1428] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:16:15.980574 5088 heartbeater.cc:503] Master 127.19.66.254:37991 was elected leader, sending a full tablet report...
I20260620 19:16:15.981144 5235 raft_consensus.cc:348] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.981288 5235 raft_consensus.cc:374] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:15.981326 5235 raft_consensus.cc:729] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351fd9e18fb746a38754e46f8e7b70a5, State: Initialized, Role: FOLLOWER
I20260620 19:16:15.981436 5235 consensus_queue.cc:260] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.981684 5235 ts_tablet_manager.cc:1428] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:16:15.981735 4957 heartbeater.cc:503] Master 127.19.66.254:37991 was elected leader, sending a full tablet report...
I20260620 19:16:15.990689 5241 raft_consensus.cc:482] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:15.990774 5241 raft_consensus.cc:504] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.991040 5241 leader_election.cc:288] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 304cf9c27d2b4255a15c34d2fee87f9d (127.19.66.195:43319), 9f960c2a35e241928cda8db7ac40d01b (127.19.66.194:46775)
I20260620 19:16:15.993757 5230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.991246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:48051 (local address 127.19.66.195:43319)
0620 19:16:15.991280 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.991281 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.991289 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:15.991334 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.991335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.991341 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.991390 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.991593 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.992104 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993082 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.993462 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993515 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.993522 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.993655 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.993697 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:15.993765 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.991193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:43319 (local address 127.19.66.193:48051)
0620 19:16:15.991236 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.991247 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.991284 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.991471 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.991472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.991474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.991536 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.991538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.992169 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.992171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.993011 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.993013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.993532 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.993597 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.993600 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.993642 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.993677 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260620 19:16:15.993794 5232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.991422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:55783 (local address 127.19.66.194:46775)
0620 19:16:15.991455 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:15.991457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:15.991464 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:15.991516 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:15.991517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:15.991525 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:15.991581 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:15.991711 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.992184 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993151 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.993486 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993546 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.993554 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:15.993708 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 19:16:15.993741 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:16:15.993989 5242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:15.991282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46775 (local address 127.19.66.193:55783)
0620 19:16:15.991422 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:15.991431 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:15.991472 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:15.991586 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:15.991587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:15.991588 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:15.991646 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.991647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.992248 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.992249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.993068 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:15.993072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:15.993559 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:15.993560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:15.993631 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:15.993635 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:15.993689 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:16:15.993722 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260620 19:16:15.994066 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a306d8024475f9b234e4ec8d41418" candidate_uuid: "351fd9e18fb746a38754e46f8e7b70a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" is_pre_election: true
I20260620 19:16:15.994212 5173 raft_consensus.cc:2454] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351fd9e18fb746a38754e46f8e7b70a5 in term 0.
I20260620 19:16:15.994267 5042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a306d8024475f9b234e4ec8d41418" candidate_uuid: "351fd9e18fb746a38754e46f8e7b70a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f960c2a35e241928cda8db7ac40d01b" is_pre_election: true
I20260620 19:16:15.994398 5042 raft_consensus.cc:2454] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 351fd9e18fb746a38754e46f8e7b70a5 in term 0.
I20260620 19:16:15.994419 4847 leader_election.cc:302] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304cf9c27d2b4255a15c34d2fee87f9d, 351fd9e18fb746a38754e46f8e7b70a5; no voters:
I20260620 19:16:15.994520 5241 raft_consensus.cc:2781] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:15.994572 5241 raft_consensus.cc:482] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:15.994587 5241 raft_consensus.cc:3037] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.995337 5241 raft_consensus.cc:504] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.995440 5241 leader_election.cc:288] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [CANDIDATE]: Term 1 election: Requested vote from peers 304cf9c27d2b4255a15c34d2fee87f9d (127.19.66.195:43319), 9f960c2a35e241928cda8db7ac40d01b (127.19.66.194:46775)
I20260620 19:16:15.995607 5042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a306d8024475f9b234e4ec8d41418" candidate_uuid: "351fd9e18fb746a38754e46f8e7b70a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f960c2a35e241928cda8db7ac40d01b"
I20260620 19:16:15.995609 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0a306d8024475f9b234e4ec8d41418" candidate_uuid: "351fd9e18fb746a38754e46f8e7b70a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "304cf9c27d2b4255a15c34d2fee87f9d"
I20260620 19:16:15.995676 5042 raft_consensus.cc:3037] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.995695 5173 raft_consensus.cc:3037] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:15.996338 5042 raft_consensus.cc:2454] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351fd9e18fb746a38754e46f8e7b70a5 in term 1.
I20260620 19:16:15.996351 5173 raft_consensus.cc:2454] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 351fd9e18fb746a38754e46f8e7b70a5 in term 1.
I20260620 19:16:15.996502 4847 leader_election.cc:302] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 304cf9c27d2b4255a15c34d2fee87f9d, 351fd9e18fb746a38754e46f8e7b70a5; no voters:
I20260620 19:16:15.996577 5241 raft_consensus.cc:2781] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:15.996727 5241 raft_consensus.cc:686] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [term 1 LEADER]: Becoming Leader. State: Replica: 351fd9e18fb746a38754e46f8e7b70a5, State: Running, Role: LEADER
I20260620 19:16:15.996811 5241 consensus_queue.cc:237] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } }
I20260620 19:16:15.997555 4769 catalog_manager.cc:5495] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 351fd9e18fb746a38754e46f8e7b70a5 (127.19.66.193). New cstate: current_term: 1 leader_uuid: "351fd9e18fb746a38754e46f8e7b70a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351fd9e18fb746a38754e46f8e7b70a5" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 41677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20260620 19:16:16.006369 5229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:16.004353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.19.66.193:41677)
0620 19:16:16.004396 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:16.004397 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:16.004404 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:16.004464 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:16.004464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:16.004480 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:16.004534 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:16.004625 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:16.005128 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:16.005799 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:16.005971 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:16.006011 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:16.006103 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:16.006203 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:16.006256 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:16.006270 (+ 14us) server_negotiation.cc:299] Negotiation successful
0620 19:16:16.006287 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:16:16.007903 5173 raft_consensus.cc:1261] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 1 FOLLOWER]: Refusing update from remote peer 351fd9e18fb746a38754e46f8e7b70a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:16.007925 5042 raft_consensus.cc:1261] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 1 FOLLOWER]: Refusing update from remote peer 351fd9e18fb746a38754e46f8e7b70a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:16.008220 5243 consensus_queue.cc:1035] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "304cf9c27d2b4255a15c34d2fee87f9d" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 43319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:16.008342 5241 consensus_queue.cc:1035] T fd0a306d8024475f9b234e4ec8d41418 P 351fd9e18fb746a38754e46f8e7b70a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f960c2a35e241928cda8db7ac40d01b" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:16.011193 5247 mvcc.cc:205] Tried to move back new op lower bound from 7299002269726236672 to 7299002269683625984. Current Snapshot: MvccSnapshot[applied={T|T < 7299002269726236672}]
I20260620 19:16:16.012593 4768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:16.012686 4768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37838:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:16.013382 4768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c14e2793cca4ac69574deb4d65430e2: Sending DeleteTablet for 3 replicas of tablet fd0a306d8024475f9b234e4ec8d41418
I20260620 19:16:16.013643 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0a306d8024475f9b234e4ec8d41418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:16 UTC) from {username='slave'} at 127.0.0.1:38704
I20260620 19:16:16.013685 5022 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0a306d8024475f9b234e4ec8d41418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:16 UTC) from {username='slave'} at 127.0.0.1:43046
I20260620 19:16:16.013872 5256 tablet_replica.cc:330] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: stopping tablet replica
I20260620 19:16:16.013939 4891 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0a306d8024475f9b234e4ec8d41418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:16 UTC) from {username='slave'} at 127.0.0.1:58638
I20260620 19:16:16.013972 5256 raft_consensus.cc:2229] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:16.014065 5256 raft_consensus.cc:2258] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:16.014214 5257 tablet_replica.cc:330] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: stopping tablet replica
I20260620 19:16:16.014297 5257 raft_consensus.cc:2229] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:16.014365 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4829
I20260620 19:16:16.014382 5257 raft_consensus.cc:2258] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:16.014899 5256 ts_tablet_manager.cc:1905] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:16.015484 5257 ts_tablet_manager.cc:1905] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:16.016556 5257 ts_tablet_manager.cc:1918] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:16.016590 5256 ts_tablet_manager.cc:1918] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:16.016616 5257 log.cc:1196] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/fd0a306d8024475f9b234e4ec8d41418
I20260620 19:16:16.016649 5256 log.cc:1196] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/fd0a306d8024475f9b234e4ec8d41418
I20260620 19:16:16.016870 5257 ts_tablet_manager.cc:1939] T fd0a306d8024475f9b234e4ec8d41418 P 304cf9c27d2b4255a15c34d2fee87f9d: Deleting consensus metadata
I20260620 19:16:16.016896 5256 ts_tablet_manager.cc:1939] T fd0a306d8024475f9b234e4ec8d41418 P 9f960c2a35e241928cda8db7ac40d01b: Deleting consensus metadata
I20260620 19:16:16.017280 4756 catalog_manager.cc:4841] TS 9f960c2a35e241928cda8db7ac40d01b (127.19.66.194:46775): tablet fd0a306d8024475f9b234e4ec8d41418 (table test-table [id=3d358f1b4a1947e694cc506d8c25ef46]) successfully deleted
I20260620 19:16:16.017364 4756 catalog_manager.cc:4841] TS 304cf9c27d2b4255a15c34d2fee87f9d (127.19.66.195:43319): tablet fd0a306d8024475f9b234e4ec8d41418 (table test-table [id=3d358f1b4a1947e694cc506d8c25ef46]) successfully deleted
W20260620 19:16:16.019917 4754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:41677 (error 108)
I20260620 19:16:16.020375 5231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:16.020250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:41677 (local address 127.0.0.1:58642)
0620 19:16:16.020298 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:16.020341 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:41677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 19:16:16.020370 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4960
W20260620 19:16:16.021190 4754 catalog_manager.cc:4584] TS 351fd9e18fb746a38754e46f8e7b70a5 (127.19.66.193:41677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd0a306d8024475f9b234e4ec8d41418: Network error: Client connection negotiation failed: client connection to 127.19.66.193:41677: connect: Connection refused (error 111)
I20260620 19:16:16.024683 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5091
I20260620 19:16:16.028299 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 4738
2026-06-20T19:16:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (627 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:16 dist-test-slave-nq8r krb5kdc[5262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:16 dist-test-slave-nq8r krb5kdc[5262](info): set up 2 sockets
Jun 20 19:16:16 dist-test-slave-nq8r krb5kdc[5262](info): commencing operation
krb5kdc: starting...
W20260620 19:16:16.972323 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:17.079493 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44667
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:17.158130 5275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:17.158332 5275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:17.158365 5275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:17.159508 5275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:17.159548 5275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:17.159567 5275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:17.159583 5275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:17.159600 5275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:17.161096 5275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:17.161331 5275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:17.161532 5275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:17.166453 5280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.166453 5283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.166445 5281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.166602 5275 server_base.cc:1034] running on GCE node
I20260620 19:16:17.166821 5275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:17.167105 5275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:17.168251 5275 hybrid_clock.cc:648] HybridClock initialized: now 1781982977168235 us; error 28 us; skew 500 ppm
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:17.170588 5275 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:17.170995 5275 webserver.cc:460] Webserver started at http://127.19.66.254:35861/ using document root <none> and password file <none>
I20260620 19:16:17.171185 5275 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:17.171226 5275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:17.171312 5275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:17.172293 5275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "9d1177ce29a146c8a59c56a17ccce882"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "058803c35578875378f2cf96fc267b90"
server_key_iv: "8efb6f83e7393b97274d37991bf5fb65"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.172583 5275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "9d1177ce29a146c8a59c56a17ccce882"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "058803c35578875378f2cf96fc267b90"
server_key_iv: "8efb6f83e7393b97274d37991bf5fb65"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.173820 5275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:17.174427 5290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.174561 5275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:17.174635 5275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "9d1177ce29a146c8a59c56a17ccce882"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "058803c35578875378f2cf96fc267b90"
server_key_iv: "8efb6f83e7393b97274d37991bf5fb65"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.174706 5275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:17.193360 5275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:17.195096 5275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:17.195214 5275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:17.198858 5275 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:44667
I20260620 19:16:17.198910 5342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:44667 every 8 connection(s)
I20260620 19:16:17.199218 5275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:17.200027 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:17.202216 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: Bootstrap starting.
I20260620 19:16:17.202725 5343 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:17.202940 5343 log.cc:824] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:17.203522 5343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: No bootstrap required, opened a new log
I20260620 19:16:17.204092 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5275
I20260620 19:16:17.204208 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:17.204387 19723 external_mini_cluster.cc:1428] Setting key 2fa229e97f52ad7952d8e5bcd60c51ba
I20260620 19:16:17.206009 5343 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } }
I20260620 19:16:17.206163 5343 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:17.206199 5343 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1177ce29a146c8a59c56a17ccce882, State: Initialized, Role: FOLLOWER
I20260620 19:16:17.206327 5343 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } }
I20260620 19:16:17.206403 5343 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:17.206439 5343 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:17.206486 5343 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:17.207096 5343 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } }
I20260620 19:16:17.207194 5343 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d1177ce29a146c8a59c56a17ccce882; no voters:
I20260620 19:16:17.207334 5343 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:17.207370 5348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:17.207468 5348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1177ce29a146c8a59c56a17ccce882, State: Running, Role: LEADER
I20260620 19:16:17.207548 5348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } }
I20260620 19:16:17.207581 5343 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:17.207846 5348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1177ce29a146c8a59c56a17ccce882. Latest consensus state: current_term: 1 leader_uuid: "9d1177ce29a146c8a59c56a17ccce882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } } }
I20260620 19:16:17.207922 5348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:17.207859 5349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d1177ce29a146c8a59c56a17ccce882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1177ce29a146c8a59c56a17ccce882" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44667 } } }
I20260620 19:16:17.208073 5349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:17.208626 5353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:17.209108 5353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:17.210737 5353 catalog_manager.cc:1347] Generated new cluster ID: c2fa1cfa634648daa5c767a23be11391
I20260620 19:16:17.210789 5353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:17.215723 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.204888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41884 (local address 127.19.66.254:44667)
0620 19:16:17.205032 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.205039 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.205071 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:16:17.205952 (+ 881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.205960 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.206011 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.206129 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:17.206852 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.208067 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.209072 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.209654 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.209762 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.212387 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:17.212391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:17.212392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:17.212403 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:17.214260 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.214644 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.214647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.214648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.214738 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.214913 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.214914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.214914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.215178 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:17.215326 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.215411 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.215530 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260620 19:16:17.220053 5353 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:17.220534 5353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:17.231213 5353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: Generated new TSK 0
I20260620 19:16:17.231391 5353 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:16:17.260123 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:17.338246 5369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:17.338439 5369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:17.338461 5369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:17.339604 5369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:17.339645 5369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:17.339679 5369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:17.341116 5369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:17.341324 5369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:17.341521 5369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:17.346120 5375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.346125 5377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.346125 5374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.346398 5369 server_base.cc:1034] running on GCE node
I20260620 19:16:17.346566 5369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:17.346760 5369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:17.347888 5369 hybrid_clock.cc:648] HybridClock initialized: now 1781982977347875 us; error 28 us; skew 500 ppm
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:17.350162 5369 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:16:17.350550 5369 webserver.cc:460] Webserver started at http://127.19.66.193:35403/ using document root <none> and password file <none>
I20260620 19:16:17.350723 5369 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:17.350760 5369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:17.350844 5369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:17.351771 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "dbd2d8ae96a94f4da6b75e6a386629c2"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "9d8bdd62c6e1980f45de7a8a532f5c6b"
server_key_iv: "aa19c74135d19f9058b48fb5f5e3c977"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.352047 5369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "dbd2d8ae96a94f4da6b75e6a386629c2"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "9d8bdd62c6e1980f45de7a8a532f5c6b"
server_key_iv: "aa19c74135d19f9058b48fb5f5e3c977"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.353210 5369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:17.353819 5384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.353962 5369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:17.354039 5369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "dbd2d8ae96a94f4da6b75e6a386629c2"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "9d8bdd62c6e1980f45de7a8a532f5c6b"
server_key_iv: "aa19c74135d19f9058b48fb5f5e3c977"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.354095 5369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:17.381448 5369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:17.383543 5369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:17.383685 5369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:17.383908 5369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:17.384189 5369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:17.384222 5369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.384249 5369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:17.384272 5369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.389657 5369 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:43623
I20260620 19:16:17.389711 5497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:43623 every 8 connection(s)
I20260620 19:16:17.390012 5369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:16:17.394732 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5369
I20260620 19:16:17.394833 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:17.395002 19723 external_mini_cluster.cc:1428] Setting key b7a1f748eccbb2256ff450a079057641
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:17.398561 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.390726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:37317 (local address 127.19.66.254:44667)
0620 19:16:17.390787 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.390790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.390964 (+ 174us) server_negotiation.cc:407] Connection header received
0620 19:16:17.391415 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.391416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.391425 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.391475 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:17.392330 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.392831 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.393785 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.393985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.394035 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.396001 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:17.396003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:17.396003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:17.396012 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:17.397287 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.397715 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.397715 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.397716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.397758 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.397984 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.397985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.397985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.398108 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:17.398172 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.398385 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.398465 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:16:17.398648 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.390682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44667 (local address 127.19.66.193:37317)
0620 19:16:17.390855 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.390890 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.391342 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.391531 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.391534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.392058 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:17.392254 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.392259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.392919 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.392920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.393713 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.393716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.394043 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.394043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.394135 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.394544 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:17.394552 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:17.395916 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:17.397355 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.397362 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.397364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.397637 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.397806 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.397807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.397807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.397918 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.398162 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:17.398167 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:17.398279 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.398356 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.398481 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260620 19:16:17.399132 5498 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44667
I20260620 19:16:17.399235 5498 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:17.399400 5498 heartbeater.cc:511] Master 127.19.66.254:44667 requested a full tablet report, sending...
I20260620 19:16:17.399905 5307 ts_manager.cc:194] Registered new tserver with Master: dbd2d8ae96a94f4da6b75e6a386629c2 (127.19.66.193:43623)
I20260620 19:16:17.400501 5307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:37317
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:16:17.416713 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:17.495260 5503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:17.495471 5503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:17.495492 5503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:17.496608 5503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:17.496650 5503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:17.496675 5503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:17.498062 5503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:17.498308 5503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:17.498574 5503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:17.503764 5511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.503767 5508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.503886 5503 server_base.cc:1034] running on GCE node
W20260620 19:16:17.503767 5509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.504197 5503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:17.504423 5503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:17.505695 5503 hybrid_clock.cc:648] HybridClock initialized: now 1781982977505680 us; error 25 us; skew 500 ppm
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:17.508123 5503 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:16:17.508550 5503 webserver.cc:460] Webserver started at http://127.19.66.194:36931/ using document root <none> and password file <none>
I20260620 19:16:17.508775 5503 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:17.508823 5503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:17.508970 5503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:17.509995 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "e143f70cc8d44f338a44d9d15082f3db"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "da3eb02de0e9e95313beaaeb0d2ed273"
server_key_iv: "ee2ac89eea1c0a770e4e1f8eb0333a70"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.510331 5503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "e143f70cc8d44f338a44d9d15082f3db"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "da3eb02de0e9e95313beaaeb0d2ed273"
server_key_iv: "ee2ac89eea1c0a770e4e1f8eb0333a70"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.511595 5503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:17.512300 5518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.512439 5503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:17.512517 5503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "e143f70cc8d44f338a44d9d15082f3db"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "da3eb02de0e9e95313beaaeb0d2ed273"
server_key_iv: "ee2ac89eea1c0a770e4e1f8eb0333a70"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.512583 5503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:17.526511 5503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:17.528455 5503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:17.528579 5503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:17.528769 5503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:17.529044 5503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:17.529076 5503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.529104 5503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:17.529135 5503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.534379 5503 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:46309
I20260620 19:16:17.534437 5631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:46309 every 8 connection(s)
I20260620 19:16:17.534698 5503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:17.541002 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5503
I20260620 19:16:17.541110 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:17.541317 19723 external_mini_cluster.cc:1428] Setting key f0149a07cac3c379399480c12704f859
I20260620 19:16:17.543259 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.535317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:46395 (local address 127.19.66.254:44667)
0620 19:16:17.535356 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.535358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.535539 (+ 181us) server_negotiation.cc:407] Connection header received
0620 19:16:17.536028 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.536030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.536042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.536089 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:17.537053 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.537543 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.538510 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.538700 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.538754 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.540641 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:17.540643 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:17.540643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:17.540650 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:17.542011 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.542369 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.542370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.542370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.542432 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.542639 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.542639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.542640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.542754 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:17.542813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.543088 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.543170 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:16:17.543383 5634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.535295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44667 (local address 127.19.66.194:46395)
0620 19:16:17.535421 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.535457 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.535924 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.536167 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.536171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.536722 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:17.536964 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.536970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.537619 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.537620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.538412 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.538415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.538765 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.538766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.538893 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.539283 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:17.539290 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:17.540558 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:17.542079 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.542086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.542089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.542303 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.542494 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.542494 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.542495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.542591 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.542810 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:17.542815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:17.542955 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.543059 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.543173 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260620 19:16:17.543789 5632 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44667
I20260620 19:16:17.543874 5632 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:17.544051 5632 heartbeater.cc:511] Master 127.19.66.254:44667 requested a full tablet report, sending...
I20260620 19:16:17.544385 5307 ts_manager.cc:194] Registered new tserver with Master: e143f70cc8d44f338a44d9d15082f3db (127.19.66.194:46309)
I20260620 19:16:17.544740 5307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:46395
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:16:17.563097 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:17.641122 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:17.641309 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:17.641330 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:17.642431 5637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:17.642469 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:17.642493 5637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:17.643886 5637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44667
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:17.644083 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:17.644286 5637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:17.648921 5642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.648926 5645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.648926 5643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.649139 5637 server_base.cc:1034] running on GCE node
I20260620 19:16:17.649327 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:17.649530 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:17.650660 5637 hybrid_clock.cc:648] HybridClock initialized: now 1781982977650642 us; error 40 us; skew 500 ppm
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:17.653048 5637 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:16:17.653463 5637 webserver.cc:460] Webserver started at http://127.19.66.195:42333/ using document root <none> and password file <none>
I20260620 19:16:17.653666 5637 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:17.653708 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:17.653887 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:17.654906 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "123f549fa0c04678939b9624994fbb99"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "e0ca62f70571edd064e924623ff91b71"
server_key_iv: "2dcf76cf52d2b16d1ef4d4a68f601773"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.655212 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "123f549fa0c04678939b9624994fbb99"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "e0ca62f70571edd064e924623ff91b71"
server_key_iv: "2dcf76cf52d2b16d1ef4d4a68f601773"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.656345 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:17.657064 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.657194 5637 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:17.657267 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "123f549fa0c04678939b9624994fbb99"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "e0ca62f70571edd064e924623ff91b71"
server_key_iv: "2dcf76cf52d2b16d1ef4d4a68f601773"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.657330 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:17.670711 5637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:17.672291 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:17.672408 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:17.672596 5637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:17.672953 5637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:17.672989 5637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.673018 5637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:17.673043 5637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:17.678509 5637 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:42021
I20260620 19:16:17.678563 5765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:42021 every 8 connection(s)
I20260620 19:16:17.678853 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
Jun 20 19:16:17 dist-test-slave-nq8r krb5kdc[5262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982977, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:17.687122 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.679484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:47675 (local address 127.19.66.254:44667)
0620 19:16:17.679559 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.679561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.679755 (+ 194us) server_negotiation.cc:407] Connection header received
0620 19:16:17.680204 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.680205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.680215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.680260 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:17.681172 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.681657 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.682568 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.682752 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.682796 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.684802 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:17.684804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:17.684804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:17.684811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:17.686083 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.686446 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.686446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.686447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.686493 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.686664 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.686670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.686670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.686761 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:17.686803 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.687005 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.687059 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 19:16:17.687281 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.679464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44667 (local address 127.19.66.195:47675)
0620 19:16:17.679605 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.679674 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.680125 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.680319 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.680323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.680899 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:17.681096 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.681101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.681720 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.681721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.682509 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.682511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.682837 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.682839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.682957 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.683389 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:17.683397 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:17.684724 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:17.686160 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.686166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.686168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.686385 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.686534 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:17.686535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:17.686535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:17.686624 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:17.686801 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:17.686803 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:17.686910 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.686984 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.687096 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260620 19:16:17.687340 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5637
I20260620 19:16:17.687434 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:17.687608 19723 external_mini_cluster.cc:1428] Setting key cae048dd2f5bc7fa4ec30e4815d3315b
I20260620 19:16:17.687764 5766 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44667
I20260620 19:16:17.687844 5766 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:17.688019 5766 heartbeater.cc:511] Master 127.19.66.254:44667 requested a full tablet report, sending...
I20260620 19:16:17.688378 5307 ts_manager.cc:194] Registered new tserver with Master: 123f549fa0c04678939b9624994fbb99 (127.19.66.195:42021)
I20260620 19:16:17.688735 5307 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:47675
I20260620 19:16:17.699141 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:17.705819 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.700448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41890 (local address 127.19.66.254:44667)
0620 19:16:17.700496 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.700498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.700606 (+ 108us) server_negotiation.cc:407] Connection header received
0620 19:16:17.700664 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.700669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.700678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.700725 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:17.701263 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.701738 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.702420 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.702586 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.702627 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.703736 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:17.703737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:17.703738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:17.703743 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:17.704972 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.705283 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.705284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.705284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.705330 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:17.705462 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:17.705462 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:17.705462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:17.705569 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:17.705637 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.705714 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.705760 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:16:17.707281 5307 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:17.708892 5307 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:17.714699 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.711091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.19.66.193:43623)
0620 19:16:17.711244 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.711248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.711257 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:16:17.711292 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.711294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.711318 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.711393 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:17.711504 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712615 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.713547 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714270 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714388 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714454 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.714576 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.714617 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260620 19:16:17.714697 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.710913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43623 (local address 127.0.0.1:54708)
0620 19:16:17.711073 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.711081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.711123 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.711402 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.711403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.711406 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:17.711444 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.711445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712704 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.712705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.713468 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.713470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714402 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.714477 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714490 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.714557 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.714608 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":2,"thread_start_us":34,"threads_started":1}
I20260620 19:16:17.714977 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.710844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.19.66.194:46309)
0620 19:16:17.710999 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.711004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.711016 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:16:17.711065 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.711068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.711083 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.711156 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:17.711309 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712492 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.713876 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714402 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714463 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714519 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.714739 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.714893 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260620 19:16:17.715160 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.710737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46309 (local address 127.0.0.1:37874)
0620 19:16:17.710893 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.710931 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.710987 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.711169 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.711172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.711174 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:17.711243 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.711244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712599 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.712600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.713787 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.713790 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714549 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.714611 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714614 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.714657 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.714689 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":29,"thread_start_us":56,"threads_started":1}
I20260620 19:16:17.715157 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.710830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.19.66.195:42021)
0620 19:16:17.711288 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.711292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.711301 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:16:17.711334 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.711336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.711349 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.711414 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:17.711576 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712380 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.713934 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714604 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714676 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714728 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.715040 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.715081 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":81,"threads_started":1}
I20260620 19:16:17.715337 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.710738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:42021 (local address 127.0.0.1:38576)
0620 19:16:17.710985 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.710996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.711046 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.711491 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.711491 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.711493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:17.711525 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.711526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.712468 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.712469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.713864 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.713866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.714861 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.714862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.714923 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.714927 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.714968 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.715273 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":5,"thread_start_us":51,"threads_started":1}
I20260620 19:16:17.715433 5432 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 (DEFAULT_TABLE table=test-table [id=ba302322b00f4285af5549476b099672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:17.715818 5432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7eac9b6a4e4d85930424bc6e9f1153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:17.715977 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 (DEFAULT_TABLE table=test-table [id=ba302322b00f4285af5549476b099672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:17.716284 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7eac9b6a4e4d85930424bc6e9f1153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:17.719075 5780 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: Bootstrap starting.
I20260620 19:16:17.719672 5780 tablet_bootstrap.cc:654] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:17.719635 5566 tablet_service.cc:1467] Processing CreateTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 (DEFAULT_TABLE table=test-table [id=ba302322b00f4285af5549476b099672]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:17.719908 5566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a7eac9b6a4e4d85930424bc6e9f1153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:17.720115 5780 log.cc:824] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:17.720753 5780 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: No bootstrap required, opened a new log
I20260620 19:16:17.720808 5780 ts_tablet_manager.cc:1397] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:17.722124 5780 raft_consensus.cc:348] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.722302 5780 raft_consensus.cc:374] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:17.722344 5780 raft_consensus.cc:729] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd2d8ae96a94f4da6b75e6a386629c2, State: Initialized, Role: FOLLOWER
I20260620 19:16:17.722476 5783 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Bootstrap starting.
I20260620 19:16:17.722484 5780 consensus_queue.cc:260] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.722553 5781 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Bootstrap starting.
I20260620 19:16:17.722747 5780 ts_tablet_manager.cc:1428] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:17.722893 5498 heartbeater.cc:503] Master 127.19.66.254:44667 was elected leader, sending a full tablet report...
I20260620 19:16:17.723047 5783 tablet_bootstrap.cc:654] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:17.723148 5781 tablet_bootstrap.cc:654] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:17.723544 5783 log.cc:824] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:17.723625 5781 log.cc:824] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:17.724203 5783 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: No bootstrap required, opened a new log
I20260620 19:16:17.724255 5783 ts_tablet_manager.cc:1397] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:17.725123 5781 tablet_bootstrap.cc:492] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: No bootstrap required, opened a new log
I20260620 19:16:17.725188 5781 ts_tablet_manager.cc:1397] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:16:17.725582 5783 raft_consensus.cc:348] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.725708 5783 raft_consensus.cc:374] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:17.725729 5783 raft_consensus.cc:729] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e143f70cc8d44f338a44d9d15082f3db, State: Initialized, Role: FOLLOWER
I20260620 19:16:17.725832 5783 consensus_queue.cc:260] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.726068 5783 ts_tablet_manager.cc:1428] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:17.726087 5632 heartbeater.cc:503] Master 127.19.66.254:44667 was elected leader, sending a full tablet report...
I20260620 19:16:17.726788 5781 raft_consensus.cc:348] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.726950 5781 raft_consensus.cc:374] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:17.726994 5781 raft_consensus.cc:729] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 123f549fa0c04678939b9624994fbb99, State: Initialized, Role: FOLLOWER
I20260620 19:16:17.727106 5781 consensus_queue.cc:260] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.727329 5781 ts_tablet_manager.cc:1428] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:17.727361 5766 heartbeater.cc:503] Master 127.19.66.254:44667 was elected leader, sending a full tablet report...
W20260620 19:16:17.785223 5633 tablet.cc:2369] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:16:17.821632 5788 raft_consensus.cc:482] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:17.821762 5788 raft_consensus.cc:504] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.822113 5788 leader_election.cc:288] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e143f70cc8d44f338a44d9d15082f3db (127.19.66.194:46309), dbd2d8ae96a94f4da6b75e6a386629c2 (127.19.66.193:43623)
I20260620 19:16:17.824723 5768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.822287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43623 (local address 127.19.66.195:39853)
0620 19:16:17.822354 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.822366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.822410 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.822550 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.822552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.822554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:17.822621 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.822622 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.823233 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.823234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.823953 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.823954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.824479 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.824549 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.824552 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.824596 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.824651 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 19:16:17.824720 5779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.822361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:39853 (local address 127.19.66.193:43623)
0620 19:16:17.822408 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.822410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.822422 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:16:17.822458 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.822459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.822470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.822534 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:17.822685 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.823146 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824040 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.824412 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824472 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.824481 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.824609 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.824649 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:16:17.824945 5777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.822387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:40627 (local address 127.19.66.194:46309)
0620 19:16:17.822426 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.822428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.822618 (+ 190us) server_negotiation.cc:407] Connection header received
0620 19:16:17.822653 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.822654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.822666 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.822715 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:17.822855 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.823334 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824214 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.824578 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824638 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.824645 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.824805 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.824868 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:16:17.825140 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.822285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46309 (local address 127.19.66.195:40627)
0620 19:16:17.822550 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.822558 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:17.822603 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:17.822729 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:17.822729 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:17.822731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:17.822797 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.822799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.823406 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.823407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.824131 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:17.824133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.824658 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.824659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:17.824719 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.824722 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:17.825024 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 19:16:17.825063 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":81,"thread_start_us":114,"threads_started":1}
I20260620 19:16:17.825093 5452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7eac9b6a4e4d85930424bc6e9f1153" candidate_uuid: "123f549fa0c04678939b9624994fbb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" is_pre_election: true
I20260620 19:16:17.825246 5452 raft_consensus.cc:2454] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f549fa0c04678939b9624994fbb99 in term 0.
I20260620 19:16:17.825475 5653 leader_election.cc:302] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f549fa0c04678939b9624994fbb99, dbd2d8ae96a94f4da6b75e6a386629c2; no voters:
I20260620 19:16:17.825455 5586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7eac9b6a4e4d85930424bc6e9f1153" candidate_uuid: "123f549fa0c04678939b9624994fbb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e143f70cc8d44f338a44d9d15082f3db" is_pre_election: true
I20260620 19:16:17.825575 5586 raft_consensus.cc:2454] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123f549fa0c04678939b9624994fbb99 in term 0.
I20260620 19:16:17.825599 5788 raft_consensus.cc:2781] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:17.825660 5788 raft_consensus.cc:482] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:17.825683 5788 raft_consensus.cc:3037] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:17.826359 5788 raft_consensus.cc:504] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.826479 5788 leader_election.cc:288] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [CANDIDATE]: Term 1 election: Requested vote from peers e143f70cc8d44f338a44d9d15082f3db (127.19.66.194:46309), dbd2d8ae96a94f4da6b75e6a386629c2 (127.19.66.193:43623)
I20260620 19:16:17.826650 5452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7eac9b6a4e4d85930424bc6e9f1153" candidate_uuid: "123f549fa0c04678939b9624994fbb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2"
I20260620 19:16:17.826675 5586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a7eac9b6a4e4d85930424bc6e9f1153" candidate_uuid: "123f549fa0c04678939b9624994fbb99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e143f70cc8d44f338a44d9d15082f3db"
I20260620 19:16:17.826730 5586 raft_consensus.cc:3037] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:17.826725 5452 raft_consensus.cc:3037] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:17.827389 5586 raft_consensus.cc:2454] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f549fa0c04678939b9624994fbb99 in term 1.
I20260620 19:16:17.827406 5452 raft_consensus.cc:2454] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123f549fa0c04678939b9624994fbb99 in term 1.
I20260620 19:16:17.827549 5656 leader_election.cc:302] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123f549fa0c04678939b9624994fbb99, e143f70cc8d44f338a44d9d15082f3db; no voters:
I20260620 19:16:17.827649 5788 raft_consensus.cc:2781] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:17.827808 5788 raft_consensus.cc:686] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 1 LEADER]: Becoming Leader. State: Replica: 123f549fa0c04678939b9624994fbb99, State: Running, Role: LEADER
I20260620 19:16:17.827888 5788 consensus_queue.cc:237] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } }
I20260620 19:16:17.828655 5306 catalog_manager.cc:5495] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 reported cstate change: term changed from 0 to 1, leader changed from <none> to 123f549fa0c04678939b9624994fbb99 (127.19.66.195). New cstate: current_term: 1 leader_uuid: "123f549fa0c04678939b9624994fbb99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "123f549fa0c04678939b9624994fbb99" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 42021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 } health_report { overall_health: UNKNOWN } } }
I20260620 19:16:17.857671 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.855539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.19.66.195:42021)
0620 19:16:17.855578 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:17.855580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:17.855614 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:16:17.855685 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:17.855686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:17.855701 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:17.855751 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:17.855844 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.856318 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.857057 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:17.857249 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:17.857299 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:17.857390 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:17.857472 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:17.857547 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:17.857584 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 19:16:17.857593 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:16:17.859303 5586 raft_consensus.cc:1261] T 5a7eac9b6a4e4d85930424bc6e9f1153 P e143f70cc8d44f338a44d9d15082f3db [term 1 FOLLOWER]: Refusing update from remote peer 123f549fa0c04678939b9624994fbb99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:17.859591 5452 raft_consensus.cc:1261] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 1 FOLLOWER]: Refusing update from remote peer 123f549fa0c04678939b9624994fbb99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:17.859638 5790 consensus_queue.cc:1035] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e143f70cc8d44f338a44d9d15082f3db" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:17.859829 5790 consensus_queue.cc:1035] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbd2d8ae96a94f4da6b75e6a386629c2" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:17.862074 5795 mvcc.cc:205] Tried to move back new op lower bound from 7299002277309427712 to 7299002277183623168. Current Snapshot: MvccSnapshot[applied={T|T < 7299002277309427712}]
I20260620 19:16:17.862458 5794 mvcc.cc:205] Tried to move back new op lower bound from 7299002277309427712 to 7299002277183623168. Current Snapshot: MvccSnapshot[applied={T|T < 7299002277309427712}]
I20260620 19:16:17.863797 5307 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:17.863874 5307 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41890:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:17.864486 5307 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d1177ce29a146c8a59c56a17ccce882: Sending DeleteTablet for 3 replicas of tablet 5a7eac9b6a4e4d85930424bc6e9f1153
I20260620 19:16:17.864804 5432 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:54708
I20260620 19:16:17.864816 5566 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:37874
I20260620 19:16:17.865090 5803 tablet_replica.cc:330] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2: stopping tablet replica
I20260620 19:16:17.865072 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a7eac9b6a4e4d85930424bc6e9f1153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:17 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:38576
I20260620 19:16:17.865159 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5369
I20260620 19:16:17.865182 5803 raft_consensus.cc:2229] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:17.865276 5803 raft_consensus.cc:2258] T 5a7eac9b6a4e4d85930424bc6e9f1153 P dbd2d8ae96a94f4da6b75e6a386629c2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 19:16:17.869841 5291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:43623 (error 108)
I20260620 19:16:17.870257 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.870137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:43623 (local address 127.0.0.1:54720)
0620 19:16:17.870184 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.870225 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 19:16:17.870312 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5503
I20260620 19:16:17.870318 5805 tablet_replica.cc:330] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: stopping tablet replica
I20260620 19:16:17.870412 5805 raft_consensus.cc:2229] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:16:17.870499 5805 raft_consensus.cc:2258] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:17.870924 5805 ts_tablet_manager.cc:1905] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:17.872511 5805 ts_tablet_manager.cc:1918] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:17.872579 5805 log.cc:1196] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/5a7eac9b6a4e4d85930424bc6e9f1153
I20260620 19:16:17.872825 5805 ts_tablet_manager.cc:1939] T 5a7eac9b6a4e4d85930424bc6e9f1153 P 123f549fa0c04678939b9624994fbb99: Deleting consensus metadata
W20260620 19:16:17.873353 5291 catalog_manager.cc:4584] TS dbd2d8ae96a94f4da6b75e6a386629c2 (127.19.66.193:43623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a7eac9b6a4e4d85930424bc6e9f1153: Network error: Client connection negotiation failed: client connection to 127.19.66.193:43623: connect: Connection refused (error 111)
I20260620 19:16:17.873461 5291 catalog_manager.cc:4841] TS 123f549fa0c04678939b9624994fbb99 (127.19.66.195:42021): tablet 5a7eac9b6a4e4d85930424bc6e9f1153 (table test-table [id=ba302322b00f4285af5549476b099672]) successfully deleted
I20260620 19:16:17.876236 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:17.876130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46309 (local address 127.0.0.1:37884)
0620 19:16:17.876180 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:17.876208 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.194:46309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:16:17.876237 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5637
I20260620 19:16:17.880417 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5275
2026-06-20T19:16:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1852 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T19:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:17Z Disabled control of system clock
I20260620 19:16:17.905710 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40613
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:40613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:17.982851 5810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:17.983052 5810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:17.983070 5810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:17.984151 5810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:17.984184 5810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:17.984193 5810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:17.984205 5810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:17.984220 5810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:17.985510 5810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:40613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:40613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:17.985690 5810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:17.985919 5810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:17.990213 5815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.990274 5818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:17.990221 5816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:17.990505 5810 server_base.cc:1034] running on GCE node
I20260620 19:16:17.990731 5810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:17.991004 5810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:17.992139 5810 hybrid_clock.cc:648] HybridClock initialized: now 1781982977992127 us; error 27 us; skew 500 ppm
I20260620 19:16:17.997650 5810 webserver.cc:460] Webserver started at http://127.19.66.254:34781/ using document root <none> and password file <none>
I20260620 19:16:17.997884 5810 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:17.997933 5810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:17.998054 5810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:17.999099 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "304e9093fa8cf2544b02a4f5d191e377"
server_key_iv: "9481e21478110a271142ac87e9074ef6"
server_key_version: "encryptionkey@0"
I20260620 19:16:17.999436 5810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "304e9093fa8cf2544b02a4f5d191e377"
server_key_iv: "9481e21478110a271142ac87e9074ef6"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.000584 5810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 19:16:18.001288 5824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.001431 5810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.001516 5810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24"
format_stamp: "Formatted at 2026-06-20 19:16:17 on dist-test-slave-nq8r"
server_key: "304e9093fa8cf2544b02a4f5d191e377"
server_key_iv: "9481e21478110a271142ac87e9074ef6"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.001582 5810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:18.014782 5810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:18.015095 5810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:18.015230 5810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:18.018797 5810 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:40613
I20260620 19:16:18.018841 5876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:40613 every 8 connection(s)
I20260620 19:16:18.019160 5810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:18.019465 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5810
I20260620 19:16:18.019562 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:18.019788 19723 external_mini_cluster.cc:1428] Setting key 1a64bab9d0a6d87e61288edffbbbc95d
I20260620 19:16:18.020251 5877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:18.023351 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: Bootstrap starting.
I20260620 19:16:18.023972 5877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:18.024281 5877 log.cc:824] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:18.025081 5877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: No bootstrap required, opened a new log
I20260620 19:16:18.025445 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.020272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.19.66.254:40613)
0620 19:16:18.020497 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.020505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.020533 (+ 28us) server_negotiation.cc:407] Connection header received
0620 19:16:18.021048 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.021057 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.021141 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.021283 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:18.021437 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.022716 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.023430 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.024017 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.024148 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.024753 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:18.024757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:18.024758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:18.024978 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:18.025019 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:18.025034 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:18.025096 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.025165 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.025247 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260620 19:16:18.026945 5877 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } }
I20260620 19:16:18.027073 5877 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:18.027094 5877 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd6c2d8b4fb14b6ebbcee51fb2967f24, State: Initialized, Role: FOLLOWER
I20260620 19:16:18.027256 5877 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } }
I20260620 19:16:18.027323 5877 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:18.027345 5877 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:18.027400 5877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:18.027896 5877 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } }
I20260620 19:16:18.027980 5877 leader_election.cc:302] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd6c2d8b4fb14b6ebbcee51fb2967f24; no voters:
I20260620 19:16:18.028138 5877 leader_election.cc:288] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:18.028162 5882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:18.028273 5882 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [term 1 LEADER]: Becoming Leader. State: Replica: dd6c2d8b4fb14b6ebbcee51fb2967f24, State: Running, Role: LEADER
I20260620 19:16:18.028358 5882 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } }
I20260620 19:16:18.028388 5877 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:18.028680 5884 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd6c2d8b4fb14b6ebbcee51fb2967f24. Latest consensus state: current_term: 1 leader_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } } }
I20260620 19:16:18.028702 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd6c2d8b4fb14b6ebbcee51fb2967f24" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 40613 } } }
I20260620 19:16:18.028841 5884 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:18.028870 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:18.029783 5892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:18.030232 5892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:18.031538 5892 catalog_manager.cc:1347] Generated new cluster ID: f2b3e156df07474f998dfcb90bb6959a
I20260620 19:16:18.031579 5892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:18.043968 5892 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:18.044505 5892 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:18.046538 5892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: Generated new TSK 0
I20260620 19:16:18.046685 5892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:18.056041 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--builtin_ntp_servers=127.19.66.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:18.154482 5901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:18.154699 5901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:18.154719 5901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:18.155851 5901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:18.155890 5901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:18.155913 5901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:18.157328 5901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:18.157543 5901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:18.157820 5901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:18.163264 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:18.163288 5907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:18.163455 5901 server_base.cc:1034] running on GCE node
W20260620 19:16:18.163338 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:18.163745 5901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:18.163970 5901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:18.165097 5901 hybrid_clock.cc:648] HybridClock initialized: now 1781982978165081 us; error 30 us; skew 500 ppm
I20260620 19:16:18.166262 5901 webserver.cc:460] Webserver started at http://127.19.66.193:33007/ using document root <none> and password file <none>
I20260620 19:16:18.166468 5901 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:18.166515 5901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:18.166636 5901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:18.167622 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "0ff1a038e10e4183a848ebf14932fce0"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "c2143cbaa332fafc4b3e9d08af77b563"
server_key_iv: "ef9fdb2cb1d3d73bea9ba5b3d16e2ea4"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.167929 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "0ff1a038e10e4183a848ebf14932fce0"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "c2143cbaa332fafc4b3e9d08af77b563"
server_key_iv: "ef9fdb2cb1d3d73bea9ba5b3d16e2ea4"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.169144 5901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:18.169785 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.169919 5901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:18.169994 5901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "0ff1a038e10e4183a848ebf14932fce0"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "c2143cbaa332fafc4b3e9d08af77b563"
server_key_iv: "ef9fdb2cb1d3d73bea9ba5b3d16e2ea4"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.170051 5901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:18.181402 5901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:18.181658 5901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:18.181772 5901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:18.181957 5901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:18.182243 5901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:18.182276 5901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.182303 5901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:18.182327 5901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.187616 5901 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:44521
I20260620 19:16:18.187678 6028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:44521 every 8 connection(s)
I20260620 19:16:18.187945 5901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:16:18.191040 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 5901
I20260620 19:16:18.191118 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:18.191274 19723 external_mini_cluster.cc:1428] Setting key e83e16908918d0d66114b722855d9f49
I20260620 19:16:18.192116 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--builtin_ntp_servers=127.19.66.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:16:18.192826 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.188546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:60053 (local address 127.19.66.254:40613)
0620 19:16:18.188590 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.188593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.188830 (+ 237us) server_negotiation.cc:407] Connection header received
0620 19:16:18.189317 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.189318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.189331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.189382 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:18.189989 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.190463 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.191451 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.191697 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.191753 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.192587 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:18.192589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:18.192590 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:18.192594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:18.192609 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:18.192614 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:18.192669 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.192718 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.192753 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:16:18.193058 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.188539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40613 (local address 127.19.66.193:60053)
0620 19:16:18.188725 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.188756 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.189242 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.189444 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.189447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.189456 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:18.189930 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.189935 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.190525 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.190526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.191381 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.191384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.191774 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.191775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.191926 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.192462 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:18.192463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:18.192490 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:18.192511 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:18.192670 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:18.192674 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:18.192676 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.192719 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.192792 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260620 19:16:18.193495 6029 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40613
I20260620 19:16:18.193588 6029 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:18.193768 6029 heartbeater.cc:511] Master 127.19.66.254:40613 requested a full tablet report, sending...
I20260620 19:16:18.194209 5841 ts_manager.cc:194] Registered new tserver with Master: 0ff1a038e10e4183a848ebf14932fce0 (127.19.66.193:44521)
I20260620 19:16:18.194783 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.193:60053
W20260620 19:16:18.269703 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:18.269889 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:18.269909 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:18.271008 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:18.271044 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:18.271070 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:18.272449 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:18.272663 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:18.272956 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:18.277459 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:18.277508 6040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:18.277498 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:18.277680 6032 server_base.cc:1034] running on GCE node
I20260620 19:16:18.277872 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:18.278080 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:18.279215 6032 hybrid_clock.cc:648] HybridClock initialized: now 1781982978279203 us; error 27 us; skew 500 ppm
I20260620 19:16:18.280313 6032 webserver.cc:460] Webserver started at http://127.19.66.194:39195/ using document root <none> and password file <none>
I20260620 19:16:18.280498 6032 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:18.280536 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:18.280628 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:18.281684 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "cc337b0d65944a81bf41952e8ab69a17"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "33a7156c542d12786371707847f02ab9"
server_key_iv: "5ee9cf83191ee759d6adf7b5d65429a0"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.281958 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "cc337b0d65944a81bf41952e8ab69a17"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "33a7156c542d12786371707847f02ab9"
server_key_iv: "5ee9cf83191ee759d6adf7b5d65429a0"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.283054 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:16:18.283667 6046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.283805 6032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:18.283883 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "cc337b0d65944a81bf41952e8ab69a17"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "33a7156c542d12786371707847f02ab9"
server_key_iv: "5ee9cf83191ee759d6adf7b5d65429a0"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.283952 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:18.296918 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:18.297219 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:18.297341 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:18.297549 6032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:18.297866 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:18.297904 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.297933 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:18.297951 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.303267 6032 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:33567
I20260620 19:16:18.303339 6159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:33567 every 8 connection(s)
I20260620 19:16:18.303617 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:16:18.306648 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6032
I20260620 19:16:18.306751 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:18.306923 19723 external_mini_cluster.cc:1428] Setting key 198d3f467e073852495b5a526dda0093
I20260620 19:16:18.307895 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--builtin_ntp_servers=127.19.66.212:44803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:16:18.308934 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.304309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:44663 (local address 127.19.66.254:40613)
0620 19:16:18.304363 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.304366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.304609 (+ 243us) server_negotiation.cc:407] Connection header received
0620 19:16:18.305146 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.305147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.305158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.305213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:18.305900 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.306383 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.307563 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.307792 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.307841 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.308685 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:18.308687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:18.308687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:18.308691 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:18.308705 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:18.308709 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:18.308743 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.308824 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.308873 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:16:18.309167 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.304302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40613 (local address 127.19.66.194:44663)
0620 19:16:18.304495 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.304530 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.305049 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.305283 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.305287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.305295 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:18.305826 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.305831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.306444 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.306445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.307476 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.307479 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.307869 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.307870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.308032 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.308528 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:18.308529 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:18.308573 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:18.308601 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:18.308754 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:18.308757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:18.308760 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.308826 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.308952 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260620 19:16:18.309597 6160 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40613
I20260620 19:16:18.309676 6160 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:18.309845 6160 heartbeater.cc:511] Master 127.19.66.254:40613 requested a full tablet report, sending...
I20260620 19:16:18.310189 5841 ts_manager.cc:194] Registered new tserver with Master: cc337b0d65944a81bf41952e8ab69a17 (127.19.66.194:33567)
I20260620 19:16:18.310626 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.194:44663
W20260620 19:16:18.386489 6163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:18.386680 6163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:18.386700 6163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:18.387809 6163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:18.387844 6163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:18.387867 6163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:18.389169 6163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:44803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:40613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:18.389340 6163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:18.389560 6163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:18.394043 6168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:18.394038 6169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:18.394035 6171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:18.394215 6163 server_base.cc:1034] running on GCE node
I20260620 19:16:18.394423 6163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:18.394666 6163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:18.395809 6163 hybrid_clock.cc:648] HybridClock initialized: now 1781982978395796 us; error 29 us; skew 500 ppm
I20260620 19:16:18.396955 6163 webserver.cc:460] Webserver started at http://127.19.66.195:43321/ using document root <none> and password file <none>
I20260620 19:16:18.397161 6163 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:18.397207 6163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:18.397327 6163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:18.398361 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "e5d06721ab304292b1d5ecc388ef30c6"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "88caecf62a97eabe9253f75f925f62ca"
server_key_iv: "ded143664f0c72aced83576579ca096a"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.398694 6163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "e5d06721ab304292b1d5ecc388ef30c6"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "88caecf62a97eabe9253f75f925f62ca"
server_key_iv: "ded143664f0c72aced83576579ca096a"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.399834 6163 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:18.400426 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.400568 6163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:18.400666 6163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "e5d06721ab304292b1d5ecc388ef30c6"
format_stamp: "Formatted at 2026-06-20 19:16:18 on dist-test-slave-nq8r"
server_key: "88caecf62a97eabe9253f75f925f62ca"
server_key_iv: "ded143664f0c72aced83576579ca096a"
server_key_version: "encryptionkey@0"
I20260620 19:16:18.400722 6163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:18.417477 6163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:18.417780 6163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:18.417908 6163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:18.418118 6163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:18.418431 6163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:18.418465 6163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.418545 6163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:18.418570 6163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:18.423789 6163 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:33453
I20260620 19:16:18.423840 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:33453 every 8 connection(s)
I20260620 19:16:18.424135 6163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:16:18.429147 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.424804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:34193 (local address 127.19.66.254:40613)
0620 19:16:18.424871 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.424872 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.425103 (+ 231us) server_negotiation.cc:407] Connection header received
0620 19:16:18.425599 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.425601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.425614 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.425664 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:18.426322 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.426819 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.427776 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.427989 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.428054 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.428883 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:18.428884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:18.428885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:18.428890 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:18.428905 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:18.428910 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:18.428951 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.429028 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.429055 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 19:16:18.429406 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.424792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:40613 (local address 127.19.66.195:34193)
0620 19:16:18.424994 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.425027 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.425505 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.425734 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.425737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.425746 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:18.426234 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.426239 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.426901 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.426902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.427694 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.427697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.428077 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.428078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.428202 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.428725 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:18.428726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:18.428756 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:18.428779 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:18.428957 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:18.428962 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:18.428966 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.429028 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.429150 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260620 19:16:18.429925 6291 heartbeater.cc:347] Connected to a master server at 127.19.66.254:40613
I20260620 19:16:18.430008 6291 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:18.430166 6291 heartbeater.cc:511] Master 127.19.66.254:40613 requested a full tablet report, sending...
I20260620 19:16:18.430521 5841 ts_manager.cc:194] Registered new tserver with Master: e5d06721ab304292b1d5ecc388ef30c6 (127.19.66.195:33453)
I20260620 19:16:18.430871 5841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.195:34193
I20260620 19:16:18.433125 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6163
I20260620 19:16:18.433207 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:18.433383 19723 external_mini_cluster.cc:1428] Setting key a2e0c6dc00bdc094b879dd75b87548e0
I20260620 19:16:18.434195 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:18.438256 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.435442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.19.66.254:40613)
0620 19:16:18.435494 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.435496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.435594 (+ 98us) server_negotiation.cc:407] Connection header received
0620 19:16:18.435662 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.435663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.435672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.435723 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:18.435863 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.436338 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.437130 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.437328 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.437378 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.437990 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:18.437992 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:18.437992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:18.437997 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:18.438011 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:18.438015 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:18.438095 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.438166 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.438176 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:16:18.439563 5841 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:18.441061 5841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:18.446746 6299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.442912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33453 (local address 127.0.0.1:58900)
0620 19:16:18.443143 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.443153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.443199 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.443464 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.443465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.443467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:18.443511 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.443512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444757 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.444758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.445816 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.445817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446467 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.446541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446553 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.446623 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.446681 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260620 19:16:18.446754 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.443006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58900 (local address 127.19.66.195:33453)
0620 19:16:18.443291 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.443295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.443303 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:18.443337 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.443339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.443351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.443413 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:18.443583 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444679 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.445902 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446394 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446456 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446553 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.446635 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.446672 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":67,"threads_started":1}
I20260620 19:16:18.447125 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.443087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:33567 (local address 127.0.0.1:42606)
0620 19:16:18.443437 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.443445 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.443489 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.443719 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.443720 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.443721 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:18.443752 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.443753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444556 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.444557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.445808 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.445810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446755 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.446816 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446980 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.447022 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.447054 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":84,"threads_started":1}
I20260620 19:16:18.447126 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.442887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44521 (local address 127.0.0.1:54748)
0620 19:16:18.443081 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.443123 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.443190 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.443399 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.443402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.443405 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:18.443478 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.443480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444956 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.444957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.445776 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.445779 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446756 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.446941 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446944 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.446999 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.447037 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260620 19:16:18.447407 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.443294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.19.66.194:33567)
0620 19:16:18.443594 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.443598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.443607 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:16:18.443638 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.443640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.443652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.443713 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:18.443801 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444486 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.445883 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446568 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446634 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446718 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.447311 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.447349 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":65,"threads_started":1}
I20260620 19:16:18.447402 6225 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 (DEFAULT_TABLE table=test-table [id=5974a61fbb3149db8e396a848cdc4cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:18.447755 6225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1aa80c2e284bb48e20cffb5e2986f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:18.448002 6094 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 (DEFAULT_TABLE table=test-table [id=5974a61fbb3149db8e396a848cdc4cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:18.448333 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1aa80c2e284bb48e20cffb5e2986f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:18.449038 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.442978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.19.66.193:44521)
0620 19:16:18.443198 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.443203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.443214 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:16:18.443290 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.443292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.443308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.443386 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:18.443541 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.444896 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.445862 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.446567 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.446634 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.446723 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.448915 (+ 2192us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.448962 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260620 19:16:18.449618 5963 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 (DEFAULT_TABLE table=test-table [id=5974a61fbb3149db8e396a848cdc4cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:18.449952 5963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1aa80c2e284bb48e20cffb5e2986f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:18.450462 6305 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Bootstrap starting.
I20260620 19:16:18.451035 6305 tablet_bootstrap.cc:654] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:18.451323 6306 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Bootstrap starting.
I20260620 19:16:18.451344 6305 log.cc:824] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:18.451881 6306 tablet_bootstrap.cc:654] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:18.452064 6305 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: No bootstrap required, opened a new log
I20260620 19:16:18.452129 6305 ts_tablet_manager.cc:1397] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:18.452183 6306 log.cc:824] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:18.452939 6306 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: No bootstrap required, opened a new log
I20260620 19:16:18.453003 6306 ts_tablet_manager.cc:1397] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:18.452929 6309 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: Bootstrap starting.
I20260620 19:16:18.453641 6309 tablet_bootstrap.cc:654] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:18.453888 6309 log.cc:824] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:18.454450 6309 tablet_bootstrap.cc:492] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: No bootstrap required, opened a new log
I20260620 19:16:18.454505 6309 ts_tablet_manager.cc:1397] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 19:16:18.454918 6305 raft_consensus.cc:348] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.455094 6305 raft_consensus.cc:374] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:18.455128 6305 raft_consensus.cc:729] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5d06721ab304292b1d5ecc388ef30c6, State: Initialized, Role: FOLLOWER
I20260620 19:16:18.455282 6305 consensus_queue.cc:260] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.455405 6306 raft_consensus.cc:348] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.455547 6305 ts_tablet_manager.cc:1428] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:16:18.455591 6291 heartbeater.cc:503] Master 127.19.66.254:40613 was elected leader, sending a full tablet report...
I20260620 19:16:18.455827 6306 raft_consensus.cc:374] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:18.455881 6306 raft_consensus.cc:729] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc337b0d65944a81bf41952e8ab69a17, State: Initialized, Role: FOLLOWER
I20260620 19:16:18.455991 6306 consensus_queue.cc:260] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.456218 6306 ts_tablet_manager.cc:1428] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 19:16:18.456264 6160 heartbeater.cc:503] Master 127.19.66.254:40613 was elected leader, sending a full tablet report...
I20260620 19:16:18.456875 6309 raft_consensus.cc:348] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.457023 6309 raft_consensus.cc:374] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:18.457059 6309 raft_consensus.cc:729] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ff1a038e10e4183a848ebf14932fce0, State: Initialized, Role: FOLLOWER
I20260620 19:16:18.457191 6309 consensus_queue.cc:260] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.457394 6309 ts_tablet_manager.cc:1428] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:16:18.457456 6029 heartbeater.cc:503] Master 127.19.66.254:40613 was elected leader, sending a full tablet report...
W20260620 19:16:18.554147 6161 tablet.cc:2369] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:16:18.629688 6312 raft_consensus.cc:482] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:18.629832 6312 raft_consensus.cc:504] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.630152 6312 leader_election.cc:288] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ff1a038e10e4183a848ebf14932fce0 (127.19.66.193:44521), e5d06721ab304292b1d5ecc388ef30c6 (127.19.66.195:33453)
I20260620 19:16:18.633699 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.630407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:38735 (local address 127.19.66.193:44521)
0620 19:16:18.630460 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.630462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.630470 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:18.630493 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.630493 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.630502 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.630549 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:18.630706 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.631431 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.632800 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.633363 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.633429 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.633437 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.633593 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.633628 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 19:16:18.633700 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.630374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:47783 (local address 127.19.66.195:33453)
0620 19:16:18.630415 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.630417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.630570 (+ 153us) server_negotiation.cc:407] Connection header received
0620 19:16:18.630603 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.630604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.630610 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.630651 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:18.630787 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.631484 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.632798 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.633363 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.633429 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.633437 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.633593 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.633628 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:16:18.633725 6162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.630313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44521 (local address 127.19.66.194:38735)
0620 19:16:18.630370 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.630382 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.630427 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.630565 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.630565 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.630567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:18.630636 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.630637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.631507 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.631508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.632720 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.632723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.633440 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.633441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.633530 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.633534 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.633583 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.633638 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32}
I20260620 19:16:18.633741 6314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.630321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:33453 (local address 127.19.66.194:47783)
0620 19:16:18.630504 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.630512 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:18.630559 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:18.630667 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:18.630667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:18.630669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:18.630729 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.630730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.631548 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.631548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.632734 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:18.632735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.633443 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.633443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:18.633533 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.633534 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 19:16:18.633583 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:16:18.633638 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":36,"thread_start_us":92,"threads_started":1}
I20260620 19:16:18.634074 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1aa80c2e284bb48e20cffb5e2986f8" candidate_uuid: "cc337b0d65944a81bf41952e8ab69a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff1a038e10e4183a848ebf14932fce0" is_pre_election: true
I20260620 19:16:18.634219 5983 raft_consensus.cc:2454] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc337b0d65944a81bf41952e8ab69a17 in term 0.
I20260620 19:16:18.634435 6048 leader_election.cc:302] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ff1a038e10e4183a848ebf14932fce0, cc337b0d65944a81bf41952e8ab69a17; no voters:
I20260620 19:16:18.634433 6245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1aa80c2e284bb48e20cffb5e2986f8" candidate_uuid: "cc337b0d65944a81bf41952e8ab69a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d06721ab304292b1d5ecc388ef30c6" is_pre_election: true
I20260620 19:16:18.634582 6245 raft_consensus.cc:2454] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc337b0d65944a81bf41952e8ab69a17 in term 0.
I20260620 19:16:18.634574 6312 raft_consensus.cc:2781] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:18.634619 6312 raft_consensus.cc:482] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:18.634647 6312 raft_consensus.cc:3037] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:18.635322 6312 raft_consensus.cc:504] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.635430 6312 leader_election.cc:288] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [CANDIDATE]: Term 1 election: Requested vote from peers 0ff1a038e10e4183a848ebf14932fce0 (127.19.66.193:44521), e5d06721ab304292b1d5ecc388ef30c6 (127.19.66.195:33453)
I20260620 19:16:18.635563 5983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1aa80c2e284bb48e20cffb5e2986f8" candidate_uuid: "cc337b0d65944a81bf41952e8ab69a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ff1a038e10e4183a848ebf14932fce0"
I20260620 19:16:18.635562 6245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1aa80c2e284bb48e20cffb5e2986f8" candidate_uuid: "cc337b0d65944a81bf41952e8ab69a17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5d06721ab304292b1d5ecc388ef30c6"
I20260620 19:16:18.635634 5983 raft_consensus.cc:3037] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:18.635634 6245 raft_consensus.cc:3037] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:18.636466 5983 raft_consensus.cc:2454] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc337b0d65944a81bf41952e8ab69a17 in term 1.
I20260620 19:16:18.636466 6245 raft_consensus.cc:2454] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc337b0d65944a81bf41952e8ab69a17 in term 1.
I20260620 19:16:18.636643 6049 leader_election.cc:302] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc337b0d65944a81bf41952e8ab69a17, e5d06721ab304292b1d5ecc388ef30c6; no voters:
I20260620 19:16:18.636765 6312 raft_consensus.cc:2781] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:18.636929 6312 raft_consensus.cc:686] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 1 LEADER]: Becoming Leader. State: Replica: cc337b0d65944a81bf41952e8ab69a17, State: Running, Role: LEADER
I20260620 19:16:18.637015 6312 consensus_queue.cc:237] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } }
I20260620 19:16:18.637789 5840 catalog_manager.cc:5495] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc337b0d65944a81bf41952e8ab69a17 (127.19.66.194). New cstate: current_term: 1 leader_uuid: "cc337b0d65944a81bf41952e8ab69a17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc337b0d65944a81bf41952e8ab69a17" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 33567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 } health_report { overall_health: UNKNOWN } } }
I20260620 19:16:18.674279 6304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.672155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.19.66.194:33567)
0620 19:16:18.672213 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:18.672216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:18.672235 (+ 19us) server_negotiation.cc:407] Connection header received
0620 19:16:18.672270 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:18.672271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:18.672290 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:18.672341 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:18.672440 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.672936 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.673638 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:18.673847 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:18.673896 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:18.673997 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:18.674080 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:18.674146 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:18.674192 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:16:18.674200 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 19:16:18.674621 6292 tablet.cc:2369] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:16:18.675988 6245 raft_consensus.cc:1261] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 1 FOLLOWER]: Refusing update from remote peer cc337b0d65944a81bf41952e8ab69a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:18.675990 5983 raft_consensus.cc:1261] T 4e1aa80c2e284bb48e20cffb5e2986f8 P 0ff1a038e10e4183a848ebf14932fce0 [term 1 FOLLOWER]: Refusing update from remote peer cc337b0d65944a81bf41952e8ab69a17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:18.676302 6312 consensus_queue.cc:1035] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ff1a038e10e4183a848ebf14932fce0" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 44521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:18.676434 6315 consensus_queue.cc:1035] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5d06721ab304292b1d5ecc388ef30c6" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 33453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:18.678617 6318 mvcc.cc:205] Tried to move back new op lower bound from 7299002280654450688 to 7299002280497778688. Current Snapshot: MvccSnapshot[applied={T|T < 7299002280654450688}]
I20260620 19:16:18.679867 6319 mvcc.cc:205] Tried to move back new op lower bound from 7299002280654450688 to 7299002280497778688. Current Snapshot: MvccSnapshot[applied={T|T < 7299002280654450688 or (T in {7299002280654450688})}]
I20260620 19:16:18.680719 5840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:18.680830 5840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:18.681550 5840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd6c2d8b4fb14b6ebbcee51fb2967f24: Sending DeleteTablet for 3 replicas of tablet 4e1aa80c2e284bb48e20cffb5e2986f8
I20260620 19:16:18.681849 5963 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:18 UTC) from {username='slave'} at 127.0.0.1:54748
I20260620 19:16:18.681865 6094 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:18 UTC) from {username='slave'} at 127.0.0.1:42606
I20260620 19:16:18.681900 6225 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e1aa80c2e284bb48e20cffb5e2986f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:18 UTC) from {username='slave'} at 127.0.0.1:58900
I20260620 19:16:18.682144 6330 tablet_replica.cc:330] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: stopping tablet replica
I20260620 19:16:18.682236 6330 raft_consensus.cc:2229] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:18.682332 6330 raft_consensus.cc:2258] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:18.682329 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5901
I20260620 19:16:18.683461 6330 ts_tablet_manager.cc:1905] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:18.684968 6330 ts_tablet_manager.cc:1918] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:18.685031 6330 log.cc:1196] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/4e1aa80c2e284bb48e20cffb5e2986f8
I20260620 19:16:18.685242 6330 ts_tablet_manager.cc:1939] T 4e1aa80c2e284bb48e20cffb5e2986f8 P e5d06721ab304292b1d5ecc388ef30c6: Deleting consensus metadata
I20260620 19:16:18.685352 6329 tablet_replica.cc:330] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: stopping tablet replica
I20260620 19:16:18.685433 6329 raft_consensus.cc:2229] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:16:18.685519 6329 raft_consensus.cc:2258] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:18.685629 5827 catalog_manager.cc:4841] TS e5d06721ab304292b1d5ecc388ef30c6 (127.19.66.195:33453): tablet 4e1aa80c2e284bb48e20cffb5e2986f8 (table test-table [id=5974a61fbb3149db8e396a848cdc4cf9]) successfully deleted
I20260620 19:16:18.685904 6329 ts_tablet_manager.cc:1905] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:18.687063 6329 ts_tablet_manager.cc:1918] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:18.687122 6329 log.cc:1196] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/4e1aa80c2e284bb48e20cffb5e2986f8
I20260620 19:16:18.687326 6329 ts_tablet_manager.cc:1939] T 4e1aa80c2e284bb48e20cffb5e2986f8 P cc337b0d65944a81bf41952e8ab69a17: Deleting consensus metadata
I20260620 19:16:18.687644 5826 catalog_manager.cc:4841] TS cc337b0d65944a81bf41952e8ab69a17 (127.19.66.194:33567): tablet 4e1aa80c2e284bb48e20cffb5e2986f8 (table test-table [id=5974a61fbb3149db8e396a848cdc4cf9]) successfully deleted
W20260620 19:16:18.689473 5826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:44521 (error 108)
I20260620 19:16:18.689641 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6032
I20260620 19:16:18.690284 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:18.689792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:44521 (local address 127.0.0.1:54752)
0620 19:16:18.690212 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:18.690252 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:44521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
W20260620 19:16:18.690410 5826 catalog_manager.cc:4584] TS 0ff1a038e10e4183a848ebf14932fce0 (127.19.66.193:44521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e1aa80c2e284bb48e20cffb5e2986f8: Network error: Client connection negotiation failed: client connection to 127.19.66.193:44521: connect: Connection refused (error 111)
I20260620 19:16:18.694435 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6163
I20260620 19:16:18.698112 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 5810
2026-06-20T19:16:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (816 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:18 dist-test-slave-nq8r krb5kdc[6334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:18 dist-test-slave-nq8r krb5kdc[6334](info): set up 2 sockets
Jun 20 19:16:18 dist-test-slave-nq8r krb5kdc[6334](info): commencing operation
krb5kdc: starting...
W20260620 19:16:19.652704 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:19 dist-test-slave-nq8r krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:19.756139 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:34035
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:34035
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:19.832774 6347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:19.832991 6347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:19.833019 6347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:19.834120 6347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:19.834156 6347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:19.834174 6347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:19.834192 6347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:19.834206 6347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:19.835624 6347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:34035
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:34035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:19.835857 6347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:19.836045 6347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:19.840404 6352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:19.840380 6353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:19.840471 6347 server_base.cc:1034] running on GCE node
W20260620 19:16:19.840380 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:19.840884 6347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:19.841162 6347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:19.842294 6347 hybrid_clock.cc:648] HybridClock initialized: now 1781982979842283 us; error 24 us; skew 500 ppm
Jun 20 19:16:19 dist-test-slave-nq8r krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982979, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:19.844455 6347 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:19.844823 6347 webserver.cc:460] Webserver started at http://127.19.66.254:41389/ using document root <none> and password file <none>
I20260620 19:16:19.845036 6347 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:19.845078 6347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:19.845197 6347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:19.846189 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "ad64e3f53d714659bb9ad4e8d466dda7"
format_stamp: "Formatted at 2026-06-20 19:16:19 on dist-test-slave-nq8r"
server_key: "50185a5e8b6a7b7db48215599fc6a0e1"
server_key_iv: "813177b8a8507f563eb51a95c1d93736"
server_key_version: "encryptionkey@0"
I20260620 19:16:19.846509 6347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "ad64e3f53d714659bb9ad4e8d466dda7"
format_stamp: "Formatted at 2026-06-20 19:16:19 on dist-test-slave-nq8r"
server_key: "50185a5e8b6a7b7db48215599fc6a0e1"
server_key_iv: "813177b8a8507f563eb51a95c1d93736"
server_key_version: "encryptionkey@0"
I20260620 19:16:19.847687 6347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:19.848332 6362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:19.848491 6347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:19.848570 6347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "ad64e3f53d714659bb9ad4e8d466dda7"
format_stamp: "Formatted at 2026-06-20 19:16:19 on dist-test-slave-nq8r"
server_key: "50185a5e8b6a7b7db48215599fc6a0e1"
server_key_iv: "813177b8a8507f563eb51a95c1d93736"
server_key_version: "encryptionkey@0"
I20260620 19:16:19.848620 6347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:19.860206 6347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:19.862120 6347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:19.862222 6347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:19.865662 6347 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:34035
I20260620 19:16:19.865698 6414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:34035 every 8 connection(s)
I20260620 19:16:19.865988 6347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:19.866726 6415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:19.868752 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: Bootstrap starting.
I20260620 19:16:19.869278 6415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:19.869515 6415 log.cc:824] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:19.869957 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6347
I20260620 19:16:19.870059 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:19.870218 6415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: No bootstrap required, opened a new log
I20260620 19:16:19.870249 19723 external_mini_cluster.cc:1428] Setting key 7a327074a14051579ea83f73b5ec8acb
I20260620 19:16:19.872300 6415 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } }
I20260620 19:16:19.872427 6415 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:19.872447 6415 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad64e3f53d714659bb9ad4e8d466dda7, State: Initialized, Role: FOLLOWER
I20260620 19:16:19.872550 6415 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } }
I20260620 19:16:19.872617 6415 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:19.872638 6415 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:19.872691 6415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:19.873239 6415 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } }
I20260620 19:16:19.873322 6415 leader_election.cc:302] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad64e3f53d714659bb9ad4e8d466dda7; no voters:
I20260620 19:16:19.873459 6415 leader_election.cc:288] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:19.873495 6420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:19.873603 6420 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [term 1 LEADER]: Becoming Leader. State: Replica: ad64e3f53d714659bb9ad4e8d466dda7, State: Running, Role: LEADER
I20260620 19:16:19.873685 6420 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } }
I20260620 19:16:19.873713 6415 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:19.873991 6420 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad64e3f53d714659bb9ad4e8d466dda7. Latest consensus state: current_term: 1 leader_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } } }
I20260620 19:16:19.874053 6420 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:19.874046 6422 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad64e3f53d714659bb9ad4e8d466dda7" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 34035 } } }
I20260620 19:16:19.874130 6422 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:19.874311 6426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:19.874776 6426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:19.876200 6426 catalog_manager.cc:1347] Generated new cluster ID: e8074fe11503438ba629f0c65ea51a26
I20260620 19:16:19.876248 6426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:16:19 dist-test-slave-nq8r krb5kdc[6334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:19.881429 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:19.870703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52152 (local address 127.19.66.254:34035)
0620 19:16:19.870864 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:19.870872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:19.870895 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:16:19.871410 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:19.871417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:19.871462 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:19.871564 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:19.872280 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:19.873602 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:19.874683 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:19.875228 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:19.875322 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:19.877889 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:19.877893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:19.877894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:19.877905 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:19.879751 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:19.880112 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:19.880115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:19.880115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:19.880224 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:19.880398 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:19.880399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:19.880399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:19.880691 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:19.880869 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:19.880972 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 19:16:19.881103 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260620 19:16:19.881526 6426 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:19.882194 6426 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:19.887856 6426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: Generated new TSK 0
I20260620 19:16:19.887990 6426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.19.66.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
Entry for principal kudu/127.19.66.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab.
I20260620 19:16:19.915771 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:20.009466 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:20.009691 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:20.009718 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:20.011088 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:20.011142 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:20.011191 6441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:20.012737 6441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.193.keytab
--principal=kudu/127.19.66.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:20.013025 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:20.013267 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:20.018520 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.018545 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.018600 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.018916 6441 server_base.cc:1034] running on GCE node
I20260620 19:16:20.019137 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:20.019411 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:20.020583 6441 hybrid_clock.cc:648] HybridClock initialized: now 1781982980020565 us; error 31 us; skew 500 ppm
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:20.023030 6441 init.cc:377] Logged in from keytab as kudu/127.19.66.193@KRBTEST.COM (short username kudu)
I20260620 19:16:20.023463 6441 webserver.cc:460] Webserver started at http://127.19.66.193:46617/ using document root <none> and password file <none>
I20260620 19:16:20.023675 6441 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:20.023725 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:20.023849 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:20.024899 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "55fef5e736f4fe3f4511dbd0754ac066"
server_key_iv: "7364f985e79f03385dacdbb1da0d4846"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.025255 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "55fef5e736f4fe3f4511dbd0754ac066"
server_key_iv: "7364f985e79f03385dacdbb1da0d4846"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.026427 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:20.027045 6456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.027189 6441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:20.027267 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "55fef5e736f4fe3f4511dbd0754ac066"
server_key_iv: "7364f985e79f03385dacdbb1da0d4846"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.027331 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:20.033777 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:20.035300 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:20.035418 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:20.035596 6441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:20.035876 6441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:20.035907 6441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.035931 6441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:20.035948 6441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.041282 6441 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:32869
I20260620 19:16:20.041347 6569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:32869 every 8 connection(s)
I20260620 19:16:20.041601 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.193@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:20.050093 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.042217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:51539 (local address 127.19.66.254:34035)
0620 19:16:20.042259 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.042261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.042438 (+ 177us) server_negotiation.cc:407] Connection header received
0620 19:16:20.042937 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.042939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.042952 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.042999 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.043923 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.044442 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.045472 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.045695 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.045752 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.047691 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.047692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.047692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:20.047701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.048799 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.049193 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.049195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.049195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.049248 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.049471 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.049472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.049473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.049585 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.049642 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.049900 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.049999 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:20.050231 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.042199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:34035 (local address 127.19.66.193:51539)
0620 19:16:20.042327 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.042360 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.042843 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.043079 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.043083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.043626 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:20.043824 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.043829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.044527 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.044528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.045389 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.045393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.045766 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.045767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.045878 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.046307 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:20.046315 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:20.047602 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:20.048899 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.048906 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.048908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.049112 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.049318 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.049319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.049319 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.049415 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.049638 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:20.049644 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:20.049782 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.049870 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.050020 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260620 19:16:20.050684 6570 heartbeater.cc:347] Connected to a master server at 127.19.66.254:34035
I20260620 19:16:20.050770 6570 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:20.050932 6570 heartbeater.cc:511] Master 127.19.66.254:34035 requested a full tablet report, sending...
I20260620 19:16:20.051011 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6441
I20260620 19:16:20.051138 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:20.051329 19723 external_mini_cluster.cc:1428] Setting key 7fd4dfcd1cded4156f3bf1fa5f60ea4c
I20260620 19:16:20.051537 6379 ts_manager.cc:194] Registered new tserver with Master: aaa3d1a5f82d4cb5a26d9ba10a891822 (127.19.66.193:32869)
I20260620 19:16:20.052330 6379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.193@KRBTEST.COM'} at 127.19.66.193:51539
WARNING: no policy specified for kudu/127.19.66.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
Entry for principal kudu/127.19.66.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab.
I20260620 19:16:20.074175 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:20.151731 6575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:20.151926 6575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:20.151948 6575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:20.153087 6575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:20.153123 6575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:20.153148 6575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:20.154513 6575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.194.keytab
--principal=kudu/127.19.66.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:20.154696 6575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:20.154865 6575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:20.159408 6581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.159420 6583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.159428 6580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.159683 6575 server_base.cc:1034] running on GCE node
I20260620 19:16:20.159868 6575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:20.160090 6575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:20.161235 6575 hybrid_clock.cc:648] HybridClock initialized: now 1781982980161216 us; error 29 us; skew 500 ppm
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:20.163607 6575 init.cc:377] Logged in from keytab as kudu/127.19.66.194@KRBTEST.COM (short username kudu)
I20260620 19:16:20.163985 6575 webserver.cc:460] Webserver started at http://127.19.66.194:41519/ using document root <none> and password file <none>
I20260620 19:16:20.164170 6575 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:20.164207 6575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:20.164299 6575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:20.165298 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "caee5383b3ed40bbb543bd57184ca686"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ed80eaf4c5ba65ad4cecc2e9b3bb7747"
server_key_iv: "ed5e464da8722cfad0437fc026c3d5f6"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.165573 6575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "caee5383b3ed40bbb543bd57184ca686"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ed80eaf4c5ba65ad4cecc2e9b3bb7747"
server_key_iv: "ed5e464da8722cfad0437fc026c3d5f6"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.166673 6575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:20.167258 6590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.167403 6575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:20.167482 6575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "caee5383b3ed40bbb543bd57184ca686"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ed80eaf4c5ba65ad4cecc2e9b3bb7747"
server_key_iv: "ed5e464da8722cfad0437fc026c3d5f6"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.167536 6575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:20.179177 6575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:20.180800 6575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:20.180938 6575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:20.181107 6575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:20.181348 6575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:20.181376 6575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.181396 6575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:20.181413 6575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.186709 6575 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:46255
I20260620 19:16:20.186784 6703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:46255 every 8 connection(s)
I20260620 19:16:20.187034 6575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:16:20.188740 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6575
I20260620 19:16:20.188829 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:20.189000 19723 external_mini_cluster.cc:1428] Setting key c7aac0deef904f8766c6e8c399915d6d
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.194@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:20.196375 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.187654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:56491 (local address 127.19.66.254:34035)
0620 19:16:20.187713 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.187715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.187884 (+ 169us) server_negotiation.cc:407] Connection header received
0620 19:16:20.188357 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.188358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.188368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.188416 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.189424 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.190005 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.191295 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.191577 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.191640 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.193876 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.193877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.193878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:20.193884 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.195091 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.195519 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.195520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.195520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.195564 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.195811 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.195811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.195811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.195904 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.195946 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.196252 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.196314 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.19.66.195@KRBTEST.COM; defaulting to no policy
I20260620 19:16:20.196623 6706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.187616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:34035 (local address 127.19.66.194:56491)
0620 19:16:20.187784 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.187819 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.188282 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.188467 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.188470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.189096 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:20.189344 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.189350 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.190093 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.190095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.191220 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.191223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.191650 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.191650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.191763 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.192225 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:20.192233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:20.193791 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:20.195171 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.195178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.195180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.195454 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.195614 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.195615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.195615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.195753 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.195956 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:20.195961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:20.196124 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.196242 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.196399 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":106,"threads_started":1}
I20260620 19:16:20.197117 6704 heartbeater.cc:347] Connected to a master server at 127.19.66.254:34035
I20260620 19:16:20.197219 6704 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:20.197384 6704 heartbeater.cc:511] Master 127.19.66.254:34035 requested a full tablet report, sending...
I20260620 19:16:20.197736 6379 ts_manager.cc:194] Registered new tserver with Master: caee5383b3ed40bbb543bd57184ca686 (127.19.66.194:46255)
I20260620 19:16:20.198110 6379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.194@KRBTEST.COM'} at 127.19.66.194:56491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
Entry for principal kudu/127.19.66.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab.
I20260620 19:16:20.211551 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:20.289463 6709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:20.289670 6709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:20.289710 6709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:20.290861 6709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:20.290901 6709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:20.290936 6709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:20.292367 6709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.195.keytab
--principal=kudu/127.19.66.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:34035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:20.292603 6709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:20.292804 6709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:20.297500 6717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.297516 6714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.297509 6715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.297705 6709 server_base.cc:1034] running on GCE node
I20260620 19:16:20.297919 6709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:20.298142 6709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:20.299291 6709 hybrid_clock.cc:648] HybridClock initialized: now 1781982980299280 us; error 26 us; skew 500 ppm
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:20.301743 6709 init.cc:377] Logged in from keytab as kudu/127.19.66.195@KRBTEST.COM (short username kudu)
I20260620 19:16:20.302145 6709 webserver.cc:460] Webserver started at http://127.19.66.195:33277/ using document root <none> and password file <none>
I20260620 19:16:20.302340 6709 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:20.302385 6709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:20.302507 6709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:20.303505 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "dcd101906c8e4c51acdc546ab9d20c83"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "e025bed4e9ace0a30c1c86bde21c61e0"
server_key_iv: "e896fd68a8db38b7a47ebee8090ffcae"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.303838 6709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "dcd101906c8e4c51acdc546ab9d20c83"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "e025bed4e9ace0a30c1c86bde21c61e0"
server_key_iv: "e896fd68a8db38b7a47ebee8090ffcae"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.305008 6709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:20.305673 6724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.305811 6709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:20.305888 6709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "dcd101906c8e4c51acdc546ab9d20c83"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "e025bed4e9ace0a30c1c86bde21c61e0"
server_key_iv: "e896fd68a8db38b7a47ebee8090ffcae"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.305950 6709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:20.316869 6709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:20.318856 6709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:20.318972 6709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:20.319168 6709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:20.319429 6709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:20.319460 6709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.319480 6709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:20.319504 6709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.324617 6709 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:40407
I20260620 19:16:20.324671 6837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:40407 every 8 connection(s)
I20260620 19:16:20.324975 6709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:16:20.325628 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6709
I20260620 19:16:20.325732 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:20.325909 19723 external_mini_cluster.cc:1428] Setting key ca0f94fec386ca892636ac97c8364bca
Jun 20 19:16:20 dist-test-slave-nq8r krb5kdc[6334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982980, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.195@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:20.333324 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.325642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:35471 (local address 127.19.66.254:34035)
0620 19:16:20.325682 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.325683 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.325930 (+ 247us) server_negotiation.cc:407] Connection header received
0620 19:16:20.326509 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.326510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.326518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.326552 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.327493 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.327979 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.328938 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.329110 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.329163 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.331044 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.331045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.331045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:20.331052 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.332285 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.332601 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.332602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.332602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.332647 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.332812 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.332812 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.332813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.332938 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.332987 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.333206 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.333263 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:20.333469 6840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.325626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:34035 (local address 127.19.66.195:35471)
0620 19:16:20.325802 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.325844 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.326429 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.326643 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.326648 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.327203 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:20.327422 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.327428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.328052 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.328053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.328883 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.328885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.329161 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.329161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.329253 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.329673 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:20.329682 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:20.330946 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:20.332346 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.332351 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.332353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.332543 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.332689 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:20.332689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:20.332689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:20.332772 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:20.332984 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:20.332987 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:20.333093 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.333179 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.333290 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260620 19:16:20.333840 6838 heartbeater.cc:347] Connected to a master server at 127.19.66.254:34035
I20260620 19:16:20.333917 6838 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:20.334044 6838 heartbeater.cc:511] Master 127.19.66.254:34035 requested a full tablet report, sending...
I20260620 19:16:20.334342 6379 ts_manager.cc:194] Registered new tserver with Master: dcd101906c8e4c51acdc546ab9d20c83 (127.19.66.195:40407)
I20260620 19:16:20.334671 6379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.19.66.195@KRBTEST.COM'} at 127.19.66.195:35471
I20260620 19:16:20.337400 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:20.343575 6419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.338592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52160 (local address 127.19.66.254:34035)
0620 19:16:20.338642 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.338644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.338732 (+ 88us) server_negotiation.cc:407] Connection header received
0620 19:16:20.338770 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.338771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.338779 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.338819 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.339284 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.339750 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.340413 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.340581 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.340620 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.341611 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.341612 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.341612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:20.341618 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.342800 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.343044 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.343044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.343045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.343087 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:20.343199 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:20.343200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:20.343200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:20.343319 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.343388 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.343459 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.343511 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:16:20.345007 6379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:20.346529 6379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:20.352587 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.19.66.194:46255)
0620 19:16:20.348936 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.348941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.348977 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:16:20.349063 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.349065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.349080 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.349143 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:20.349377 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350439 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.351667 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352206 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352269 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352327 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.352458 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.352498 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260620 19:16:20.352587 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46255 (local address 127.0.0.1:60446)
0620 19:16:20.348922 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.348933 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.348990 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.349145 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.349146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.349148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:20.349215 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.349218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350537 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.350539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.351602 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.351605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352277 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.352354 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352367 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.352427 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.352481 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":47,"threads_started":1}
I20260620 19:16:20.352751 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50698 (local address 127.19.66.193:32869)
0620 19:16:20.348950 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.348953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.348963 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:16:20.348996 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.348998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.349011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.349080 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:20.349255 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350054 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.351413 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352205 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352270 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352326 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.352631 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.352674 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":91,"threads_started":1}
I20260620 19:16:20.352984 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.19.66.195:40407)
0620 19:16:20.348897 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.348902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.349306 (+ 404us) server_negotiation.cc:407] Connection header received
0620 19:16:20.349347 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.349349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.349363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.349424 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:20.349596 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350649 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.351762 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352441 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352494 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352538 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.352834 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.352906 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260620 19:16:20.352777 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40407 (local address 127.0.0.1:46214)
0620 19:16:20.349198 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.349207 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.349252 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.349484 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.349485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.349486 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:20.349520 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.349521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350729 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.350730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.351667 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.351669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352570 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.352633 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352636 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.352679 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.352723 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":67,"threads_started":1}
I20260620 19:16:20.353065 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.348447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:32869 (local address 127.0.0.1:50698)
0620 19:16:20.348595 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.348624 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.348711 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.349096 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.349099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.349102 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:20.349172 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.349174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.350139 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.350140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.351344 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.351347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.352360 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.352361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.352423 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.352428 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.352474 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.352516 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260620 19:16:20.353758 6638 tablet_service.cc:1467] Processing CreateTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b (DEFAULT_TABLE table=test-table [id=363b65463ab34e84b3c7330b4de0befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:20.353785 6772 tablet_service.cc:1467] Processing CreateTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b (DEFAULT_TABLE table=test-table [id=363b65463ab34e84b3c7330b4de0befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:20.353952 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b (DEFAULT_TABLE table=test-table [id=363b65463ab34e84b3c7330b4de0befb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:20.354050 6638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:20.354099 6772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:20.354213 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:20.356470 6852 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Bootstrap starting.
I20260620 19:16:20.357142 6852 tablet_bootstrap.cc:654] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:20.357259 6854 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Bootstrap starting.
I20260620 19:16:20.357671 6852 log.cc:824] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:20.357852 6854 tablet_bootstrap.cc:654] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:20.358371 6854 log.cc:824] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:20.358456 6852 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: No bootstrap required, opened a new log
I20260620 19:16:20.358530 6852 ts_tablet_manager.cc:1397] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 19:16:20.358757 6853 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Bootstrap starting.
I20260620 19:16:20.359061 6854 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: No bootstrap required, opened a new log
I20260620 19:16:20.359119 6854 ts_tablet_manager.cc:1397] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:20.359368 6853 tablet_bootstrap.cc:654] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:20.360059 6853 log.cc:824] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:20.360437 6852 raft_consensus.cc:348] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.360599 6852 raft_consensus.cc:374] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:20.360643 6852 raft_consensus.cc:729] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caee5383b3ed40bbb543bd57184ca686, State: Initialized, Role: FOLLOWER
I20260620 19:16:20.360806 6853 tablet_bootstrap.cc:492] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: No bootstrap required, opened a new log
I20260620 19:16:20.360690 6854 raft_consensus.cc:348] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.360786 6852 consensus_queue.cc:260] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.360867 6854 raft_consensus.cc:374] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:20.360893 6853 ts_tablet_manager.cc:1397] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:20.360909 6854 raft_consensus.cc:729] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd101906c8e4c51acdc546ab9d20c83, State: Initialized, Role: FOLLOWER
I20260620 19:16:20.361030 6854 consensus_queue.cc:260] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.361123 6852 ts_tablet_manager.cc:1428] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 19:16:20.361200 6704 heartbeater.cc:503] Master 127.19.66.254:34035 was elected leader, sending a full tablet report...
I20260620 19:16:20.361289 6854 ts_tablet_manager.cc:1428] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:20.361310 6838 heartbeater.cc:503] Master 127.19.66.254:34035 was elected leader, sending a full tablet report...
I20260620 19:16:20.362603 6853 raft_consensus.cc:348] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.362726 6853 raft_consensus.cc:374] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:20.362746 6853 raft_consensus.cc:729] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa3d1a5f82d4cb5a26d9ba10a891822, State: Initialized, Role: FOLLOWER
I20260620 19:16:20.362824 6853 consensus_queue.cc:260] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.363052 6853 ts_tablet_manager.cc:1428] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 19:16:20.363077 6570 heartbeater.cc:503] Master 127.19.66.254:34035 was elected leader, sending a full tablet report...
W20260620 19:16:20.437592 6705 tablet.cc:2369] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:16:20.542232 6571 tablet.cc:2369] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 19:16:20.575497 6839 tablet.cc:2369] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 19:16:20.618482 6860 raft_consensus.cc:482] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:20.618606 6860 raft_consensus.cc:504] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.618978 6860 leader_election.cc:288] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcd101906c8e4c51acdc546ab9d20c83 (127.19.66.195:40407), caee5383b3ed40bbb543bd57184ca686 (127.19.66.194:46255)
I20260620 19:16:20.621752 6850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.619235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:34265 (local address 127.19.66.195:40407)
0620 19:16:20.619278 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.619280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.619367 (+ 87us) server_negotiation.cc:407] Connection header received
0620 19:16:20.619428 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.619429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.619439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.619487 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:20.619621 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.620086 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621068 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.621445 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621507 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.621515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.621646 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.621681 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:16:20.621798 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.619226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:54539 (local address 127.19.66.194:46255)
0620 19:16:20.619279 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.619281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.619423 (+ 142us) server_negotiation.cc:407] Connection header received
0620 19:16:20.619487 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.619488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.619497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.619541 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:20.619697 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.620178 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621114 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.621497 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621560 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.621567 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.621699 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.621735 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:16:20.621855 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.619164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:46255 (local address 127.19.66.193:54539)
0620 19:16:20.619361 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.619369 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.619427 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.619548 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.619548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.619550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:20.619636 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.619637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.620244 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.620245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.621047 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.621050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.621567 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.621637 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.621640 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.621690 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.621727 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":50,"threads_started":1}
I20260620 19:16:20.621763 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.619149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:40407 (local address 127.19.66.193:34265)
0620 19:16:20.619289 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.619300 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.619373 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.619495 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.619495 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.619497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:20.619570 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.619572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.620154 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.620155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.620992 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.620994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.621515 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.621516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.621585 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.621590 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.621636 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.621675 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":60,"threads_started":1}
I20260620 19:16:20.622159 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bbee97d8ad4b9fbc4b1e8fcbfafd8b" candidate_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caee5383b3ed40bbb543bd57184ca686" is_pre_election: true
I20260620 19:16:20.622295 6658 raft_consensus.cc:2454] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa3d1a5f82d4cb5a26d9ba10a891822 in term 0.
I20260620 19:16:20.622257 6792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bbee97d8ad4b9fbc4b1e8fcbfafd8b" candidate_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd101906c8e4c51acdc546ab9d20c83" is_pre_election: true
I20260620 19:16:20.622390 6792 raft_consensus.cc:2454] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa3d1a5f82d4cb5a26d9ba10a891822 in term 0.
I20260620 19:16:20.622491 6459 leader_election.cc:302] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa3d1a5f82d4cb5a26d9ba10a891822, caee5383b3ed40bbb543bd57184ca686; no voters:
I20260620 19:16:20.622612 6860 raft_consensus.cc:2781] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:20.622660 6860 raft_consensus.cc:482] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:20.622675 6860 raft_consensus.cc:3037] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:20.623353 6860 raft_consensus.cc:504] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.623464 6860 leader_election.cc:288] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [CANDIDATE]: Term 1 election: Requested vote from peers dcd101906c8e4c51acdc546ab9d20c83 (127.19.66.195:40407), caee5383b3ed40bbb543bd57184ca686 (127.19.66.194:46255)
I20260620 19:16:20.623617 6658 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bbee97d8ad4b9fbc4b1e8fcbfafd8b" candidate_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caee5383b3ed40bbb543bd57184ca686"
I20260620 19:16:20.623675 6792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66bbee97d8ad4b9fbc4b1e8fcbfafd8b" candidate_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcd101906c8e4c51acdc546ab9d20c83"
I20260620 19:16:20.623705 6658 raft_consensus.cc:3037] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:20.623772 6792 raft_consensus.cc:3037] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:20.624593 6658 raft_consensus.cc:2454] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa3d1a5f82d4cb5a26d9ba10a891822 in term 1.
I20260620 19:16:20.624593 6792 raft_consensus.cc:2454] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa3d1a5f82d4cb5a26d9ba10a891822 in term 1.
I20260620 19:16:20.624747 6459 leader_election.cc:302] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa3d1a5f82d4cb5a26d9ba10a891822, caee5383b3ed40bbb543bd57184ca686; no voters:
I20260620 19:16:20.624871 6860 raft_consensus.cc:2781] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:20.625012 6860 raft_consensus.cc:686] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [term 1 LEADER]: Becoming Leader. State: Replica: aaa3d1a5f82d4cb5a26d9ba10a891822, State: Running, Role: LEADER
I20260620 19:16:20.625105 6860 consensus_queue.cc:237] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } }
I20260620 19:16:20.625835 6379 catalog_manager.cc:5495] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa3d1a5f82d4cb5a26d9ba10a891822 (127.19.66.193). New cstate: current_term: 1 leader_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa3d1a5f82d4cb5a26d9ba10a891822" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 32869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 } health_report { overall_health: UNKNOWN } } }
I20260620 19:16:20.635494 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.633329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.19.66.193:32869)
0620 19:16:20.633373 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.633375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.633383 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:20.633457 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.633458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.633477 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.633538 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:20.633653 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.634103 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.634806 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.635005 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.635057 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.635203 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:20.635307 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:20.635376 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.635411 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.635430 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 19:16:20.637056 6658 raft_consensus.cc:1261] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 1 FOLLOWER]: Refusing update from remote peer aaa3d1a5f82d4cb5a26d9ba10a891822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:20.637056 6792 raft_consensus.cc:1261] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 1 FOLLOWER]: Refusing update from remote peer aaa3d1a5f82d4cb5a26d9ba10a891822: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:20.637347 6860 consensus_queue.cc:1035] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcd101906c8e4c51acdc546ab9d20c83" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:20.637471 6863 consensus_queue.cc:1035] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P aaa3d1a5f82d4cb5a26d9ba10a891822 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caee5383b3ed40bbb543bd57184ca686" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 46255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:20.639550 6866 mvcc.cc:205] Tried to move back new op lower bound from 7299002288687079424 to 7299002288640958464. Current Snapshot: MvccSnapshot[applied={T|T < 7299002288687079424}]
I20260620 19:16:20.642382 6379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:20.642465 6379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:20.643110 6379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad64e3f53d714659bb9ad4e8d466dda7: Sending DeleteTablet for 3 replicas of tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b
I20260620 19:16:20.643465 6638 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:20 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:60446
I20260620 19:16:20.643462 6504 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:20 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:50698
I20260620 19:16:20.643460 6772 tablet_service.cc:1514] Processing DeleteTablet for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:20 UTC) from {username='kudu', principal='kudu/127.19.66.254@KRBTEST.COM'} at 127.0.0.1:46214
I20260620 19:16:20.643743 6876 tablet_replica.cc:330] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: stopping tablet replica
I20260620 19:16:20.643836 6876 raft_consensus.cc:2229] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:20.643904 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6441
I20260620 19:16:20.643924 6876 raft_consensus.cc:2258] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:20.644495 6875 tablet_replica.cc:330] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: stopping tablet replica
I20260620 19:16:20.644598 6875 raft_consensus.cc:2229] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:20.644704 6875 raft_consensus.cc:2258] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:20.645577 6875 ts_tablet_manager.cc:1905] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:20.645762 6876 ts_tablet_manager.cc:1905] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:20.647024 6876 ts_tablet_manager.cc:1918] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:20.647082 6876 log.cc:1196] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/66bbee97d8ad4b9fbc4b1e8fcbfafd8b
I20260620 19:16:20.647109 6875 ts_tablet_manager.cc:1918] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:20.647181 6875 log.cc:1196] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/66bbee97d8ad4b9fbc4b1e8fcbfafd8b
I20260620 19:16:20.647339 6876 ts_tablet_manager.cc:1939] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P caee5383b3ed40bbb543bd57184ca686: Deleting consensus metadata
I20260620 19:16:20.647418 6875 ts_tablet_manager.cc:1939] T 66bbee97d8ad4b9fbc4b1e8fcbfafd8b P dcd101906c8e4c51acdc546ab9d20c83: Deleting consensus metadata
I20260620 19:16:20.647710 6365 catalog_manager.cc:4841] TS caee5383b3ed40bbb543bd57184ca686 (127.19.66.194:46255): tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b (table test-table [id=363b65463ab34e84b3c7330b4de0befb]) successfully deleted
I20260620 19:16:20.647744 6363 catalog_manager.cc:4841] TS dcd101906c8e4c51acdc546ab9d20c83 (127.19.66.195:40407): tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b (table test-table [id=363b65463ab34e84b3c7330b4de0befb]) successfully deleted
W20260620 19:16:20.649281 6363 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:32869 (error 108)
I20260620 19:16:20.649415 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6575
I20260620 19:16:20.650233 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.650120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:32869 (local address 127.0.0.1:50718)
0620 19:16:20.650170 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.650205 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:32869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 19:16:20.650336 6363 catalog_manager.cc:4584] TS aaa3d1a5f82d4cb5a26d9ba10a891822 (127.19.66.193:32869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 66bbee97d8ad4b9fbc4b1e8fcbfafd8b: Network error: Client connection negotiation failed: client connection to 127.19.66.193:32869: connect: Connection refused (error 111)
I20260620 19:16:20.653505 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6709
I20260620 19:16:20.658380 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6347
2026-06-20T19:16:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1960 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T19:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:20Z Disabled control of system clock
I20260620 19:16:20.683463 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44057
--webserver_interface=127.19.66.254
--webserver_port=0
--builtin_ntp_servers=127.19.66.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:20.760166 6882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:20.760385 6882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:20.760413 6882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:20.761583 6882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:20.761627 6882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:20.761644 6882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:20.761660 6882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:20.761683 6882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:20.763023 6882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:44057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:20.763268 6882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:20.763532 6882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:20.767989 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.768059 6882 server_base.cc:1034] running on GCE node
W20260620 19:16:20.767995 6888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.768002 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.768452 6882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:20.768734 6882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:20.769865 6882 hybrid_clock.cc:648] HybridClock initialized: now 1781982980769851 us; error 30 us; skew 500 ppm
I20260620 19:16:20.771028 6882 webserver.cc:460] Webserver started at http://127.19.66.254:43591/ using document root <none> and password file <none>
I20260620 19:16:20.771251 6882 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:20.771299 6882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:20.771431 6882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:20.772423 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "15b231d1b9d2457db300e9c3f9ba4ae2"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "1f38a43046f0d7b84124c8f9f32942da"
server_key_iv: "99e03f8789c0d963d11c160563a6a32e"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.772742 6882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "15b231d1b9d2457db300e9c3f9ba4ae2"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "1f38a43046f0d7b84124c8f9f32942da"
server_key_iv: "99e03f8789c0d963d11c160563a6a32e"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.773875 6882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:20.774555 6896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.774706 6882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:20.774793 6882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "15b231d1b9d2457db300e9c3f9ba4ae2"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "1f38a43046f0d7b84124c8f9f32942da"
server_key_iv: "99e03f8789c0d963d11c160563a6a32e"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.774852 6882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:20.785806 6882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:20.786073 6882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:20.786201 6882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:20.789762 6882 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:44057
I20260620 19:16:20.789803 6948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:44057 every 8 connection(s)
I20260620 19:16:20.790107 6882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:20.791028 6949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:20.793378 6949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: Bootstrap starting.
I20260620 19:16:20.793920 6949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:20.794184 6949 log.cc:824] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:20.794821 6949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: No bootstrap required, opened a new log
I20260620 19:16:20.796361 6949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } }
I20260620 19:16:20.796494 6949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:20.796515 6949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b231d1b9d2457db300e9c3f9ba4ae2, State: Initialized, Role: FOLLOWER
I20260620 19:16:20.796656 6949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } }
I20260620 19:16:20.796717 6949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:20.796736 6949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:20.796765 6949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:20.797048 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6882
I20260620 19:16:20.797145 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:20.797313 19723 external_mini_cluster.cc:1428] Setting key 35128e1a6cdafd926b0ee2d3d90368f0
I20260620 19:16:20.797328 6949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } }
I20260620 19:16:20.797420 6949 leader_election.cc:302] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15b231d1b9d2457db300e9c3f9ba4ae2; no voters:
I20260620 19:16:20.797572 6949 leader_election.cc:288] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:20.797663 6952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:20.797814 6949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:20.797827 6952 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [term 1 LEADER]: Becoming Leader. State: Replica: 15b231d1b9d2457db300e9c3f9ba4ae2, State: Running, Role: LEADER
I20260620 19:16:20.799201 6952 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } }
I20260620 19:16:20.799939 6955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } } }
I20260620 19:16:20.800002 6955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:20.799722 6953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15b231d1b9d2457db300e9c3f9ba4ae2. Latest consensus state: current_term: 1 leader_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15b231d1b9d2457db300e9c3f9ba4ae2" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44057 } } }
W20260620 19:16:20.800091 6968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 19:16:20.800145 6953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2 [sys.catalog]: This master's current role is: LEADER
W20260620 19:16:20.800165 6968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:20.800968 6971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:20.801134 6971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:20.802443 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.797733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60852 (local address 127.19.66.254:44057)
0620 19:16:20.797940 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.797949 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.797982 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:16:20.798111 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.798117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.798184 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.798341 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.798539 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.799602 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.800607 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.801049 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.801167 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.801759 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.801763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.801763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:20.801956 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.801998 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:20.802012 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.802084 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.802160 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.802238 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":111,"threads_started":1}
I20260620 19:16:20.802866 6971 catalog_manager.cc:1347] Generated new cluster ID: 713e5d09a8fd4b90a32ed66ebd320bb2
I20260620 19:16:20.802914 6971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:20.814901 6971 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:20.815560 6971 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:20.823546 6971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: Generated new TSK 0
I20260620 19:16:20.823743 6971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:20.828960 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.193:0
--local_ip_for_outbound_sockets=127.19.66.193
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--builtin_ntp_servers=127.19.66.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:20.906603 6973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:20.906798 6973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:20.906826 6973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:20.907919 6973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:20.907958 6973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:20.907989 6973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.193
I20260620 19:16:20.909343 6973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.19.66.193
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.193
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:20.909575 6973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:20.909857 6973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:20.915455 6978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.915480 6981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:20.915478 6979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:20.915663 6973 server_base.cc:1034] running on GCE node
I20260620 19:16:20.915921 6973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:20.916193 6973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:20.917340 6973 hybrid_clock.cc:648] HybridClock initialized: now 1781982980917326 us; error 29 us; skew 500 ppm
I20260620 19:16:20.918550 6973 webserver.cc:460] Webserver started at http://127.19.66.193:36309/ using document root <none> and password file <none>
I20260620 19:16:20.918763 6973 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:20.918809 6973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:20.918934 6973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:20.919966 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data/instance:
uuid: "c125edb17b47418f9f6ec19a4980185d"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ea2a5fa752eb2bd069c30b91d58061d5"
server_key_iv: "aa9432636c10bf1db3d3732d67156a58"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.920310 6973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance:
uuid: "c125edb17b47418f9f6ec19a4980185d"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ea2a5fa752eb2bd069c30b91d58061d5"
server_key_iv: "aa9432636c10bf1db3d3732d67156a58"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.921597 6973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:20.922331 6987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.922487 6973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:20.922566 6973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal
uuid: "c125edb17b47418f9f6ec19a4980185d"
format_stamp: "Formatted at 2026-06-20 19:16:20 on dist-test-slave-nq8r"
server_key: "ea2a5fa752eb2bd069c30b91d58061d5"
server_key_iv: "aa9432636c10bf1db3d3732d67156a58"
server_key_version: "encryptionkey@0"
I20260620 19:16:20.922636 6973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:20.934784 6973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:20.935063 6973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:20.935194 6973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:20.935388 6973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:20.935688 6973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:20.935722 6973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.935750 6973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:20.935773 6973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:20.941148 6973 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.193:40823
I20260620 19:16:20.941205 7100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.193:40823 every 8 connection(s)
I20260620 19:16:20.941501 6973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/data/info.pb
I20260620 19:16:20.943239 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 6973
I20260620 19:16:20.943315 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-0/wal/instance
I20260620 19:16:20.943467 19723 external_mini_cluster.cc:1428] Setting key c000758d78c101fa43e921bbffaa4bff
I20260620 19:16:20.944350 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.194:0
--local_ip_for_outbound_sockets=127.19.66.194
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--builtin_ntp_servers=127.19.66.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 19:16:20.946759 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.942130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.193:46815 (local address 127.19.66.254:44057)
0620 19:16:20.942203 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:20.942205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:20.942413 (+ 208us) server_negotiation.cc:407] Connection header received
0620 19:16:20.942914 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:20.942915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:20.942928 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:20.942968 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:20.943666 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.944225 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.945361 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.945582 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.945635 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.946496 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:20.946497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:20.946498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:20.946504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:20.946520 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:20.946524 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:20.946571 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:20.946642 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:16:20.946678 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 19:16:20.947007 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:20.942123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44057 (local address 127.19.66.193:46815)
0620 19:16:20.942303 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:20.942337 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:20.942822 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:20.943023 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:20.943026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:20.943034 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:20.943567 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.943574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.944307 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.944308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.945252 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:20.945256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:20.945655 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:20.945656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:20.945825 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:20.946352 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:20.946353 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:20.946385 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:20.946415 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:20.946577 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:20.946582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:20.946585 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:20.946647 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:16:20.946761 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260620 19:16:20.947438 7101 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44057
I20260620 19:16:20.947553 7101 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:20.947781 7101 heartbeater.cc:511] Master 127.19.66.254:44057 requested a full tablet report, sending...
I20260620 19:16:20.948318 6913 ts_manager.cc:194] Registered new tserver with Master: c125edb17b47418f9f6ec19a4980185d (127.19.66.193:40823)
I20260620 19:16:20.948999 6913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.193:46815
W20260620 19:16:21.024299 7104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:21.024493 7104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:21.024516 7104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:21.025712 7104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:21.025754 7104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:21.025779 7104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.194
I20260620 19:16:21.027119 7104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.19.66.194
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.194
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:21.027335 7104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:21.027602 7104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:21.032140 7110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:21.032193 7104 server_base.cc:1034] running on GCE node
W20260620 19:16:21.032140 7112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:21.032120 7109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:21.032495 7104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:21.032706 7104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:21.033838 7104 hybrid_clock.cc:648] HybridClock initialized: now 1781982981033823 us; error 31 us; skew 500 ppm
I20260620 19:16:21.034931 7104 webserver.cc:460] Webserver started at http://127.19.66.194:44903/ using document root <none> and password file <none>
I20260620 19:16:21.035132 7104 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:21.035187 7104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:21.035305 7104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:21.036334 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data/instance:
uuid: "be957e4ff0ad445db087a94e46ba1c0f"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "027c7bf933255e226779fc477197a040"
server_key_iv: "61b6bd652c5d5a1399c59fd72260c02f"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.036669 7104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance:
uuid: "be957e4ff0ad445db087a94e46ba1c0f"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "027c7bf933255e226779fc477197a040"
server_key_iv: "61b6bd652c5d5a1399c59fd72260c02f"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.037887 7104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:16:21.038534 7118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.038683 7104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:21.038761 7104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal
uuid: "be957e4ff0ad445db087a94e46ba1c0f"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "027c7bf933255e226779fc477197a040"
server_key_iv: "61b6bd652c5d5a1399c59fd72260c02f"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.038822 7104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:21.045339 7104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:21.045583 7104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:21.045696 7104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:21.045871 7104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:21.046157 7104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:21.046190 7104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.046218 7104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:21.046241 7104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.051563 7104 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.194:39789
I20260620 19:16:21.051614 7231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.194:39789 every 8 connection(s)
I20260620 19:16:21.051898 7104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/data/info.pb
I20260620 19:16:21.056439 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.052511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:37551 (local address 127.19.66.254:44057)
0620 19:16:21.052563 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.052565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.052787 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:16:21.053319 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.053320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.053331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.053389 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:21.054013 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.054480 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.055376 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.055545 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.055590 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.056244 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:21.056245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:21.056245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:21.056249 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:21.056263 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:21.056267 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:21.056303 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.056359 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.056382 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:16:21.056609 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.052485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44057 (local address 127.19.66.194:37551)
0620 19:16:21.052676 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.052713 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.053213 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.053450 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.053453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.053461 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:21.053952 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.053957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.054537 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.054538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.055316 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.055318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.055601 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.055601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.055715 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.056132 (+ 417us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:21.056132 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:21.056160 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:21.056181 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:21.056308 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:21.056310 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:21.056312 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.056357 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.056428 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260620 19:16:21.057053 7232 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44057
I20260620 19:16:21.057132 7232 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:21.057305 7232 heartbeater.cc:511] Master 127.19.66.254:44057 requested a full tablet report, sending...
I20260620 19:16:21.057608 6913 ts_manager.cc:194] Registered new tserver with Master: be957e4ff0ad445db087a94e46ba1c0f (127.19.66.194:39789)
I20260620 19:16:21.057940 6913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.194:37551
I20260620 19:16:21.058993 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7104
I20260620 19:16:21.059064 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal/instance
I20260620 19:16:21.059202 19723 external_mini_cluster.cc:1428] Setting key 285651d3190f74084d53d66d5bbd8a6a
I20260620 19:16:21.059940 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.19.66.195:0
--local_ip_for_outbound_sockets=127.19.66.195
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--builtin_ntp_servers=127.19.66.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 19:16:21.138234 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:21.138429 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:21.138451 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:21.139578 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:21.139614 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:21.139638 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.19.66.195
I20260620 19:16:21.140954 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.19.66.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.19.66.195
--webserver_port=0
--tserver_master_addrs=127.19.66.254:44057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.19.66.195
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:21.141180 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:21.141438 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:21.145910 7241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:21.145928 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:21.145949 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:21.146271 7235 server_base.cc:1034] running on GCE node
I20260620 19:16:21.146445 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:21.146656 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:21.147789 7235 hybrid_clock.cc:648] HybridClock initialized: now 1781982981147775 us; error 26 us; skew 500 ppm
I20260620 19:16:21.148986 7235 webserver.cc:460] Webserver started at http://127.19.66.195:34103/ using document root <none> and password file <none>
I20260620 19:16:21.149223 7235 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:21.149273 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:21.149397 7235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:21.150439 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data/instance:
uuid: "d0d58ac94a1a44cba53bde03b53639ec"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "ccc7e4feedbd73fe5c420331c75cd969"
server_key_iv: "87cb1ef1e797aba216f091b76bd4bd8a"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.150765 7235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance:
uuid: "d0d58ac94a1a44cba53bde03b53639ec"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "ccc7e4feedbd73fe5c420331c75cd969"
server_key_iv: "87cb1ef1e797aba216f091b76bd4bd8a"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.151913 7235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:21.152658 7249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.152833 7235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:21.152933 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal
uuid: "d0d58ac94a1a44cba53bde03b53639ec"
format_stamp: "Formatted at 2026-06-20 19:16:21 on dist-test-slave-nq8r"
server_key: "ccc7e4feedbd73fe5c420331c75cd969"
server_key_iv: "87cb1ef1e797aba216f091b76bd4bd8a"
server_key_version: "encryptionkey@0"
I20260620 19:16:21.152997 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:21.164822 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:21.165122 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:21.165228 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:21.165393 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 19:16:21.165680 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 19:16:21.165712 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.165740 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 19:16:21.165763 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:21.171059 7235 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.195:41045
I20260620 19:16:21.171103 7362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.195:41045 every 8 connection(s)
I20260620 19:16:21.171430 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/data/info.pb
I20260620 19:16:21.174178 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7235
I20260620 19:16:21.174260 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal/instance
I20260620 19:16:21.174435 19723 external_mini_cluster.cc:1428] Setting key e6edced4c79759d47668291bed76f343
I20260620 19:16:21.176416 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.172097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.195:40431 (local address 127.19.66.254:44057)
0620 19:16:21.172157 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.172160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.172379 (+ 219us) server_negotiation.cc:407] Connection header received
0620 19:16:21.172884 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.172885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.172896 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.172952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:21.173612 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.174093 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.175124 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.175334 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.175388 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.176132 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:21.176133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:21.176133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:21.176138 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:21.176153 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:21.176157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:21.176201 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.176305 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.176338 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 19:16:21.176599 7365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.172085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:44057 (local address 127.19.66.195:40431)
0620 19:16:21.172266 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.172301 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.172788 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.173017 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.173020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.173028 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:21.173514 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.173518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.174175 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.174176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.175044 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.175046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.175398 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.175399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.175531 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.176003 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 19:16:21.176004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:21.176033 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 19:16:21.176055 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:21.176235 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:21.176240 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:21.176242 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.176303 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.176402 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260620 19:16:21.177000 7363 heartbeater.cc:347] Connected to a master server at 127.19.66.254:44057
I20260620 19:16:21.177089 7363 heartbeater.cc:464] Registering TS with master...
I20260620 19:16:21.177259 7363 heartbeater.cc:511] Master 127.19.66.254:44057 requested a full tablet report, sending...
I20260620 19:16:21.177548 6913 ts_manager.cc:194] Registered new tserver with Master: d0d58ac94a1a44cba53bde03b53639ec (127.19.66.195:41045)
I20260620 19:16:21.177877 6913 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.19.66.195:40431
I20260620 19:16:21.186002 19723 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 19:16:21.189781 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.187274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60856 (local address 127.19.66.254:44057)
0620 19:16:21.187317 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.187319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.187435 (+ 116us) server_negotiation.cc:407] Connection header received
0620 19:16:21.187470 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.187471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.187481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.187521 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:21.187638 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.188094 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.188813 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.189029 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.189076 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.189590 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:21.189591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:21.189591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:21.189596 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:21.189608 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:21.189612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:21.189646 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.189681 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.189712 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:21.191071 6913 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 19:16:21.192636 6913 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 19:16:21.198537 7371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.194:39789 (local address 127.0.0.1:40130)
0620 19:16:21.194834 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.194874 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.194949 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.195198 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.195198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.195200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:21.195247 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.195248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196502 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.196503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.197357 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.197360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.198250 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.198251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.198341 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.198352 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.198406 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.198452 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260620 19:16:21.198572 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.19.66.194:39789)
0620 19:16:21.194882 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.194886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.195018 (+ 132us) server_negotiation.cc:407] Connection header received
0620 19:16:21.195058 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.195060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.195076 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.195138 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:21.195315 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196425 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.197438 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.198185 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.198253 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.198342 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.198430 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.198468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260620 19:16:21.198621 7372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:41045 (local address 127.0.0.1:53406)
0620 19:16:21.194858 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.194887 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.194944 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.195142 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.195145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.195147 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:21.195217 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.195218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196469 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.196470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.197572 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.197575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.198439 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.198440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.198488 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.198491 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.198537 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.198567 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":30,"thread_start_us":81,"threads_started":1}
I20260620 19:16:21.199177 7166 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecccae80a774a799356faee2b65b1a0 (DEFAULT_TABLE table=test-table [id=bf6e49f95efd42808a4d056919730fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:21.199491 7166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecccae80a774a799356faee2b65b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:21.199808 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40823 (local address 127.0.0.1:41780)
0620 19:16:21.195072 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.195081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.195125 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.195352 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.195353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.195355 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:21.195387 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.195388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196297 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.196298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.197572 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.197575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.199590 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.199591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.199652 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.199657 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.199712 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.199747 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260620 19:16:21.199882 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.19.66.193:40823)
0620 19:16:21.195026 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.195031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.195171 (+ 140us) server_negotiation.cc:407] Connection header received
0620 19:16:21.195210 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.195212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.195227 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.195293 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:21.195452 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196225 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.198990 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.199491 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.199548 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.199736 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.199762 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.199801 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":71,"threads_started":1}
I20260620 19:16:21.200026 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.194652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.19.66.195:41045)
0620 19:16:21.194965 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.194968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.194980 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:16:21.195017 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.195019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.195035 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.195127 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:21.195287 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.196391 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.197651 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.198364 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.198427 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.198516 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.198545 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.198577 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":73,"threads_started":1}
I20260620 19:16:21.200500 7035 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecccae80a774a799356faee2b65b1a0 (DEFAULT_TABLE table=test-table [id=bf6e49f95efd42808a4d056919730fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:21.200634 7297 tablet_service.cc:1467] Processing CreateTablet for tablet 0ecccae80a774a799356faee2b65b1a0 (DEFAULT_TABLE table=test-table [id=bf6e49f95efd42808a4d056919730fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 19:16:21.200829 7035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecccae80a774a799356faee2b65b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:21.200951 7297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ecccae80a774a799356faee2b65b1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:21.202255 7377 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Bootstrap starting.
I20260620 19:16:21.202775 7377 tablet_bootstrap.cc:654] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:21.203033 7377 log.cc:824] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:21.203639 7377 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: No bootstrap required, opened a new log
I20260620 19:16:21.203704 7377 ts_tablet_manager.cc:1397] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:21.205008 7380 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Bootstrap starting.
I20260620 19:16:21.205622 7380 tablet_bootstrap.cc:654] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:21.205928 7380 log.cc:824] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:21.205945 7377 raft_consensus.cc:348] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.206105 7377 raft_consensus.cc:374] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:21.206151 7377 raft_consensus.cc:729] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be957e4ff0ad445db087a94e46ba1c0f, State: Initialized, Role: FOLLOWER
I20260620 19:16:21.206278 7379 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: Bootstrap starting.
I20260620 19:16:21.206274 7377 consensus_queue.cc:260] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.206522 7377 ts_tablet_manager.cc:1428] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 19:16:21.206562 7232 heartbeater.cc:503] Master 127.19.66.254:44057 was elected leader, sending a full tablet report...
I20260620 19:16:21.206660 7380 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: No bootstrap required, opened a new log
I20260620 19:16:21.206712 7380 ts_tablet_manager.cc:1397] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 19:16:21.206830 7379 tablet_bootstrap.cc:654] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:21.207098 7379 log.cc:824] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:21.208653 7380 raft_consensus.cc:348] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.208777 7380 raft_consensus.cc:374] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:21.208806 7380 raft_consensus.cc:729] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d58ac94a1a44cba53bde03b53639ec, State: Initialized, Role: FOLLOWER
I20260620 19:16:21.208935 7380 consensus_queue.cc:260] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.209169 7380 ts_tablet_manager.cc:1428] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:16:21.209175 7363 heartbeater.cc:503] Master 127.19.66.254:44057 was elected leader, sending a full tablet report...
I20260620 19:16:21.209386 7379 tablet_bootstrap.cc:492] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: No bootstrap required, opened a new log
I20260620 19:16:21.209447 7379 ts_tablet_manager.cc:1397] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 19:16:21.211247 7379 raft_consensus.cc:348] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.211359 7379 raft_consensus.cc:374] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:21.211378 7379 raft_consensus.cc:729] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c125edb17b47418f9f6ec19a4980185d, State: Initialized, Role: FOLLOWER
I20260620 19:16:21.211460 7379 consensus_queue.cc:260] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.211687 7379 ts_tablet_manager.cc:1428] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 19:16:21.211745 7101 heartbeater.cc:503] Master 127.19.66.254:44057 was elected leader, sending a full tablet report...
I20260620 19:16:21.222061 7382 raft_consensus.cc:482] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 19:16:21.222142 7382 raft_consensus.cc:504] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.222396 7382 leader_election.cc:288] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0d58ac94a1a44cba53bde03b53639ec (127.19.66.195:41045), c125edb17b47418f9f6ec19a4980185d (127.19.66.193:40823)
I20260620 19:16:21.225965 7376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.222682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:60429 (local address 127.19.66.193:40823)
0620 19:16:21.222734 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.222735 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.222866 (+ 131us) server_negotiation.cc:407] Connection header received
0620 19:16:21.222927 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.222928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.222939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.222985 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:21.223145 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.223814 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225050 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.225621 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225687 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.225696 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.225868 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.225904 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:16:21.225984 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.222532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.195:41045 (local address 127.19.66.194:51603)
0620 19:16:21.222577 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.222588 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.222635 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.222797 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.222799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.222800 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:21.222869 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.222871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.223762 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.223763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.224971 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.224973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.225709 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.225806 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.225810 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.225869 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.225897 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":8}
I20260620 19:16:21.225991 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.222632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40823 (local address 127.19.66.194:60429)
0620 19:16:21.222802 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.222811 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:21.222869 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:21.223032 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:21.223033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:21.223034 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 19:16:21.223095 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.223096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.223873 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.223873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.224971 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:21.224973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.225705 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:21.225806 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.225810 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 19:16:21.225859 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 19:16:21.225895 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260620 19:16:21.226013 7373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.222578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.19.66.194:51603 (local address 127.19.66.195:41045)
0620 19:16:21.222622 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.222624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.222633 (+ 9us) server_negotiation.cc:407] Connection header received
0620 19:16:21.222722 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.222723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.222733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.222781 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 19:16:21.222933 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.223696 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225045 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.225621 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.225707 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.225714 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.225922 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.225960 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 19:16:21.226366 7055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecccae80a774a799356faee2b65b1a0" candidate_uuid: "be957e4ff0ad445db087a94e46ba1c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c125edb17b47418f9f6ec19a4980185d" is_pre_election: true
I20260620 19:16:21.226366 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecccae80a774a799356faee2b65b1a0" candidate_uuid: "be957e4ff0ad445db087a94e46ba1c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d58ac94a1a44cba53bde03b53639ec" is_pre_election: true
I20260620 19:16:21.226478 7055 raft_consensus.cc:2454] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be957e4ff0ad445db087a94e46ba1c0f in term 0.
I20260620 19:16:21.226511 7317 raft_consensus.cc:2454] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be957e4ff0ad445db087a94e46ba1c0f in term 0.
I20260620 19:16:21.226667 7122 leader_election.cc:302] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be957e4ff0ad445db087a94e46ba1c0f, c125edb17b47418f9f6ec19a4980185d; no voters:
I20260620 19:16:21.226783 7382 raft_consensus.cc:2781] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 19:16:21.226841 7382 raft_consensus.cc:482] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 19:16:21.226863 7382 raft_consensus.cc:3037] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:21.227495 7382 raft_consensus.cc:504] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.227617 7382 leader_election.cc:288] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [CANDIDATE]: Term 1 election: Requested vote from peers d0d58ac94a1a44cba53bde03b53639ec (127.19.66.195:41045), c125edb17b47418f9f6ec19a4980185d (127.19.66.193:40823)
I20260620 19:16:21.227733 7317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecccae80a774a799356faee2b65b1a0" candidate_uuid: "be957e4ff0ad445db087a94e46ba1c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0d58ac94a1a44cba53bde03b53639ec"
I20260620 19:16:21.227797 7317 raft_consensus.cc:3037] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:21.227787 7055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ecccae80a774a799356faee2b65b1a0" candidate_uuid: "be957e4ff0ad445db087a94e46ba1c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c125edb17b47418f9f6ec19a4980185d"
I20260620 19:16:21.227849 7055 raft_consensus.cc:3037] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:21.228425 7317 raft_consensus.cc:2454] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be957e4ff0ad445db087a94e46ba1c0f in term 1.
I20260620 19:16:21.228478 7055 raft_consensus.cc:2454] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be957e4ff0ad445db087a94e46ba1c0f in term 1.
I20260620 19:16:21.228629 7122 leader_election.cc:302] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be957e4ff0ad445db087a94e46ba1c0f, c125edb17b47418f9f6ec19a4980185d; no voters:
I20260620 19:16:21.228718 7382 raft_consensus.cc:2781] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:21.228863 7382 raft_consensus.cc:686] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 1 LEADER]: Becoming Leader. State: Replica: be957e4ff0ad445db087a94e46ba1c0f, State: Running, Role: LEADER
I20260620 19:16:21.228956 7382 consensus_queue.cc:237] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } }
I20260620 19:16:21.229729 6913 catalog_manager.cc:5495] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to be957e4ff0ad445db087a94e46ba1c0f (127.19.66.194). New cstate: current_term: 1 leader_uuid: "be957e4ff0ad445db087a94e46ba1c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be957e4ff0ad445db087a94e46ba1c0f" member_type: VOTER last_known_addr { host: "127.19.66.194" port: 39789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 } health_report { overall_health: UNKNOWN } } }
I20260620 19:16:21.234678 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.232477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.19.66.194:39789)
0620 19:16:21.232524 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:21.232525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:21.232533 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:21.232563 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:21.232564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:21.232581 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:21.232641 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:21.232751 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.233262 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.233975 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:21.234175 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:21.234228 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:21.234361 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:21.234465 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:21.234534 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:21.234551 (+ 17us) server_negotiation.cc:299] Negotiation successful
0620 19:16:21.234567 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:16:21.236505 7055 raft_consensus.cc:1261] T 0ecccae80a774a799356faee2b65b1a0 P c125edb17b47418f9f6ec19a4980185d [term 1 FOLLOWER]: Refusing update from remote peer be957e4ff0ad445db087a94e46ba1c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:21.236506 7317 raft_consensus.cc:1261] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 1 FOLLOWER]: Refusing update from remote peer be957e4ff0ad445db087a94e46ba1c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:21.236814 7382 consensus_queue.cc:1035] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c125edb17b47418f9f6ec19a4980185d" member_type: VOTER last_known_addr { host: "127.19.66.193" port: 40823 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:21.236972 7387 consensus_queue.cc:1035] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0d58ac94a1a44cba53bde03b53639ec" member_type: VOTER last_known_addr { host: "127.19.66.195" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:21.239517 7391 mvcc.cc:205] Tried to move back new op lower bound from 7299002291142156288 to 7299002291114422272. Current Snapshot: MvccSnapshot[applied={T|T < 7299002291142156288}]
I20260620 19:16:21.240442 7392 mvcc.cc:205] Tried to move back new op lower bound from 7299002291142156288 to 7299002291114422272. Current Snapshot: MvccSnapshot[applied={T|T < 7299002291142156288}]
I20260620 19:16:21.241576 6912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:21.241667 6912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 19:16:21.242360 6912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15b231d1b9d2457db300e9c3f9ba4ae2: Sending DeleteTablet for 3 replicas of tablet 0ecccae80a774a799356faee2b65b1a0
I20260620 19:16:21.242687 7297 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecccae80a774a799356faee2b65b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:21 UTC) from {username='slave'} at 127.0.0.1:53406
I20260620 19:16:21.242717 7166 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecccae80a774a799356faee2b65b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:21 UTC) from {username='slave'} at 127.0.0.1:40130
I20260620 19:16:21.242730 7035 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ecccae80a774a799356faee2b65b1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 19:16:21 UTC) from {username='slave'} at 127.0.0.1:41780
I20260620 19:16:21.242923 7400 tablet_replica.cc:330] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: stopping tablet replica
I20260620 19:16:21.243212 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6973
I20260620 19:16:21.245621 7400 raft_consensus.cc:2229] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 19:16:21.245730 7400 raft_consensus.cc:2258] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:21.246078 7400 ts_tablet_manager.cc:1905] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:21.247293 7400 ts_tablet_manager.cc:1918] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:21.247352 7400 log.cc:1196] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-2/wal/wals/0ecccae80a774a799356faee2b65b1a0
I20260620 19:16:21.247550 7400 ts_tablet_manager.cc:1939] T 0ecccae80a774a799356faee2b65b1a0 P d0d58ac94a1a44cba53bde03b53639ec: Deleting consensus metadata
I20260620 19:16:21.247901 6900 catalog_manager.cc:4841] TS d0d58ac94a1a44cba53bde03b53639ec (127.19.66.195:41045): tablet 0ecccae80a774a799356faee2b65b1a0 (table test-table [id=bf6e49f95efd42808a4d056919730fee]) successfully deleted
I20260620 19:16:21.248955 7401 tablet_replica.cc:330] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: stopping tablet replica
I20260620 19:16:21.249048 7401 raft_consensus.cc:2229] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 1 LEADER]: Raft consensus shutting down.
I20260620 19:16:21.249137 7401 raft_consensus.cc:2258] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 19:16:21.249543 7401 ts_tablet_manager.cc:1905] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 19:16:21.250990 7401 ts_tablet_manager.cc:1918] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 19:16:21.251058 7401 log.cc:1196] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Deleting WAL directory at /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781982751812665-19723-0/minicluster-data/ts-1/wal/wals/0ecccae80a774a799356faee2b65b1a0
I20260620 19:16:21.251272 7401 ts_tablet_manager.cc:1939] T 0ecccae80a774a799356faee2b65b1a0 P be957e4ff0ad445db087a94e46ba1c0f: Deleting consensus metadata
I20260620 19:16:21.251638 6899 catalog_manager.cc:4841] TS be957e4ff0ad445db087a94e46ba1c0f (127.19.66.194:39789): tablet 0ecccae80a774a799356faee2b65b1a0 (table test-table [id=bf6e49f95efd42808a4d056919730fee]) successfully deleted
W20260620 19:16:21.252676 6900 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.19.66.193:40823 (error 108)
I20260620 19:16:21.252794 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7104
I20260620 19:16:21.253566 7375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:21.253422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.193:40823 (local address 127.0.0.1:41792)
0620 19:16:21.253489 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:21.253538 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.193:40823: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 19:16:21.253672 6900 catalog_manager.cc:4584] TS c125edb17b47418f9f6ec19a4980185d (127.19.66.193:40823): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ecccae80a774a799356faee2b65b1a0: Network error: Client connection negotiation failed: client connection to 127.19.66.193:40823: connect: Connection refused (error 111)
I20260620 19:16:21.257566 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7235
I20260620 19:16:21.261485 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 6882
2026-06-20T19:16:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (602 ms)
[----------] 6 tests from EncryptionPolicyTest (7737 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:21 dist-test-slave-nq8r krb5kdc[7406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:21 dist-test-slave-nq8r krb5kdc[7406](info): set up 2 sockets
Jun 20 19:16:21 dist-test-slave-nq8r krb5kdc[7406](info): commencing operation
krb5kdc: starting...
W20260620 19:16:22.204566 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:22.310616 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:22.388688 7419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:22.388929 7419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:22.388957 7419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:22.390065 7419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:22.390100 7419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:22.390111 7419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:22.390122 7419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:22.390133 7419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:22.391520 7419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:22.391705 7419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:22.391867 7419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:22.396456 7424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:22.396445 7427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:22.396471 7419 server_base.cc:1034] running on GCE node
W20260620 19:16:22.396445 7425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:22.396912 7419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:22.397207 7419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:22.398346 7419 hybrid_clock.cc:648] HybridClock initialized: now 1781982982398330 us; error 36 us; skew 500 ppm
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:22.400575 7419 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:22.401022 7419 webserver.cc:460] Webserver started at http://127.0.0.1:33067/ using document root <none> and password file <none>
I20260620 19:16:22.401201 7419 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:22.401239 7419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:22.401329 7419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:22.402274 7419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f18a4a8c141b43959b78ed413f5f0239"
format_stamp: "Formatted at 2026-06-20 19:16:22 on dist-test-slave-nq8r"
server_key: "20e4d525bab673d4b31721b9e2f91433"
server_key_iv: "35a87166c73a35706fe30f273bb07169"
server_key_version: "encryptionkey@0"
I20260620 19:16:22.402550 7419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f18a4a8c141b43959b78ed413f5f0239"
format_stamp: "Formatted at 2026-06-20 19:16:22 on dist-test-slave-nq8r"
server_key: "20e4d525bab673d4b31721b9e2f91433"
server_key_iv: "35a87166c73a35706fe30f273bb07169"
server_key_version: "encryptionkey@0"
I20260620 19:16:22.403651 7419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:16:22.404251 7434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:22.404397 7419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:22.404475 7419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f18a4a8c141b43959b78ed413f5f0239"
format_stamp: "Formatted at 2026-06-20 19:16:22 on dist-test-slave-nq8r"
server_key: "20e4d525bab673d4b31721b9e2f91433"
server_key_iv: "35a87166c73a35706fe30f273bb07169"
server_key_version: "encryptionkey@0"
I20260620 19:16:22.404534 7419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:22.431106 7419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:22.432971 7419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:22.433116 7419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:22.436822 7419 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41559
I20260620 19:16:22.436882 7486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41559 every 8 connection(s)
I20260620 19:16:22.437224 7419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:22.438017 7487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:22.440274 7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: Bootstrap starting.
I20260620 19:16:22.440805 7487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:22.441078 7487 log.cc:824] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:22.441681 7487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: No bootstrap required, opened a new log
I20260620 19:16:22.443212 7487 raft_consensus.cc:348] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260620 19:16:22.443346 7487 raft_consensus.cc:374] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:22.443378 7487 raft_consensus.cc:729] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18a4a8c141b43959b78ed413f5f0239, State: Initialized, Role: FOLLOWER
I20260620 19:16:22.443517 7487 consensus_queue.cc:260] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260620 19:16:22.443575 7487 raft_consensus.cc:388] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:22.443598 7487 raft_consensus.cc:482] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:22.443640 7487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:22.444130 7487 raft_consensus.cc:504] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260620 19:16:22.444228 7487 leader_election.cc:302] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18a4a8c141b43959b78ed413f5f0239; no voters:
I20260620 19:16:22.444386 7487 leader_election.cc:288] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:22.444415 7490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:22.444552 7490 raft_consensus.cc:686] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [term 1 LEADER]: Becoming Leader. State: Replica: f18a4a8c141b43959b78ed413f5f0239, State: Running, Role: LEADER
I20260620 19:16:22.444605 7487 sys_catalog.cc:564] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:22.444640 7490 consensus_queue.cc:237] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } }
I20260620 19:16:22.445050 7492 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f18a4a8c141b43959b78ed413f5f0239. Latest consensus state: current_term: 1 leader_uuid: "f18a4a8c141b43959b78ed413f5f0239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260620 19:16:22.445129 7492 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:22.445190 7491 sys_catalog.cc:455] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f18a4a8c141b43959b78ed413f5f0239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18a4a8c141b43959b78ed413f5f0239" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41559 } } }
I20260620 19:16:22.445262 7491 sys_catalog.cc:458] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:22.445341 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7419
I20260620 19:16:22.445434 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:22.445616 19723 external_mini_cluster.cc:1428] Setting key 0aceff0f909c59fe993d0b93c8d33e19
W20260620 19:16:22.446576 7505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:22.446628 7505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:22.446677 7498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:22.446856 7498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:22.448195 7498 catalog_manager.cc:1347] Generated new cluster ID: dcc522312bae422184164264c530d6f4
I20260620 19:16:22.448276 7498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:16:22.455536 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:22.446178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60807 (local address 127.0.0.1:41559)
0620 19:16:22.446496 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:22.446506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:22.446538 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:16:22.446634 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:22.446641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:22.447355 (+ 714us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:22.447489 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:22.448162 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:22.449195 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:22.449915 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:22.450328 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:22.450423 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:22.452537 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:22.452542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:22.452542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:22.452554 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:22.454105 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:22.454457 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:22.454459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:22.454460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:22.454554 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:22.454757 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:22.454758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:22.454758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:22.455003 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:22.455146 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:22.455273 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 19:16:22.455373 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":84,"threads_started":1}
I20260620 19:16:22.456377 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:22.466497 7498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:22.466722 7508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:22.457737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50039 (local address 127.0.0.1:41559)
0620 19:16:22.457783 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:22.457785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:22.461042 (+ 3257us) server_negotiation.cc:407] Connection header received
0620 19:16:22.461102 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:22.461103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:22.461113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:22.461157 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:22.461681 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:22.462166 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:22.462825 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:22.463039 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:22.463094 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:22.464184 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:22.464185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:22.464185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:22.464192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:22.465445 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:22.465780 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:22.465781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:22.465782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:22.465833 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:22.466027 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:22.466028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:22.466029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:22.466183 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:22.466291 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:22.466438 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:16:22.466506 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 19:16:22.467181 7498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:22.472255 7498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f18a4a8c141b43959b78ed413f5f0239: Generated new TSK 0
I20260620 19:16:22.472429 7498 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:22.477212 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7419
W20260620 19:16:22.480859 7404 connection.cc:535] client connection to 127.0.0.1:41559 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41559): Connection reset by peer (error 104)
2026-06-20T19:16:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7518](info): set up 2 sockets
Jun 20 19:16:22 dist-test-slave-nq8r krb5kdc[7518](info): commencing operation
krb5kdc: starting...
W20260620 19:16:23.420548 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:23 dist-test-slave-nq8r krb5kdc[7518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:23.525211 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:23.604285 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:23.604494 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:23.604524 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:23.605692 7531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:23.605734 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:23.605753 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:23.605769 7531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:23.605783 7531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:23.607213 7531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:23.607451 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:23.607658 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:23.612072 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:23.612035 7539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:23.612044 7537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:23.612242 7531 server_base.cc:1034] running on GCE node
I20260620 19:16:23.612458 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:23.612761 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:23.613906 7531 hybrid_clock.cc:648] HybridClock initialized: now 1781982983613902 us; error 38 us; skew 500 ppm
Jun 20 19:16:23 dist-test-slave-nq8r krb5kdc[7518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:23.621546 7531 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:23.621992 7531 webserver.cc:460] Webserver started at http://127.0.0.1:38157/ using document root <none> and password file <none>
I20260620 19:16:23.622184 7531 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:23.622231 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:23.622354 7531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:23.623359 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "d26ba14e8ece4245b0d0c32ee880b564"
format_stamp: "Formatted at 2026-06-20 19:16:23 on dist-test-slave-nq8r"
server_key: "c120becdde2ef3d4b5b509d8002fea9e"
server_key_iv: "ef869f3141e473ff97db082ea0ceeb33"
server_key_version: "encryptionkey@0"
I20260620 19:16:23.623685 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "d26ba14e8ece4245b0d0c32ee880b564"
format_stamp: "Formatted at 2026-06-20 19:16:23 on dist-test-slave-nq8r"
server_key: "c120becdde2ef3d4b5b509d8002fea9e"
server_key_iv: "ef869f3141e473ff97db082ea0ceeb33"
server_key_version: "encryptionkey@0"
I20260620 19:16:23.624840 7531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:23.625514 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:23.625671 7531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:23.625762 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "d26ba14e8ece4245b0d0c32ee880b564"
format_stamp: "Formatted at 2026-06-20 19:16:23 on dist-test-slave-nq8r"
server_key: "c120becdde2ef3d4b5b509d8002fea9e"
server_key_iv: "ef869f3141e473ff97db082ea0ceeb33"
server_key_version: "encryptionkey@0"
I20260620 19:16:23.625828 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:23.653451 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:23.655216 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:23.655349 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:23.659013 7531 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38407
I20260620 19:16:23.659054 7598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38407 every 8 connection(s)
I20260620 19:16:23.659376 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:23.659647 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7531
I20260620 19:16:23.659750 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:23.659922 19723 external_mini_cluster.cc:1428] Setting key eb0a94e7f404d9fe9f9f23f22a05c0b4
I20260620 19:16:23.660318 7599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:23.663522 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564: Bootstrap starting.
I20260620 19:16:23.664170 7599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:23.664471 7599 log.cc:824] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:23.665216 7599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564: No bootstrap required, opened a new log
I20260620 19:16:23.665622 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:23.660399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43165 (local address 127.0.0.1:38407)
0620 19:16:23.660585 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:23.660594 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:23.660627 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:16:23.661141 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:23.661151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:23.661213 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:23.661358 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:23.661515 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:23.662828 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:23.663598 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:23.664186 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:23.664294 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:23.664956 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:23.664961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:23.664961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:23.665147 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:23.665184 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:23.665198 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:23.665283 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:23.665392 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 19:16:23.665417 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
I20260620 19:16:23.666601 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:23.667322 7599 raft_consensus.cc:348] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260620 19:16:23.667457 7599 raft_consensus.cc:374] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:23.667498 7599 raft_consensus.cc:729] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d26ba14e8ece4245b0d0c32ee880b564, State: Initialized, Role: FOLLOWER
I20260620 19:16:23.667605 7599 consensus_queue.cc:260] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260620 19:16:23.667675 7599 raft_consensus.cc:388] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:23.667703 7599 raft_consensus.cc:482] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:23.667748 7599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:23.668251 7599 raft_consensus.cc:504] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260620 19:16:23.668341 7599 leader_election.cc:302] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d26ba14e8ece4245b0d0c32ee880b564; no voters:
I20260620 19:16:23.668483 7599 leader_election.cc:288] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:23.668521 7609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:23.668622 7609 raft_consensus.cc:686] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [term 1 LEADER]: Becoming Leader. State: Replica: d26ba14e8ece4245b0d0c32ee880b564, State: Running, Role: LEADER
I20260620 19:16:23.668713 7599 sys_catalog.cc:564] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:23.668730 7609 consensus_queue.cc:237] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } }
I20260620 19:16:23.669052 7609 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d26ba14e8ece4245b0d0c32ee880b564. Latest consensus state: current_term: 1 leader_uuid: "d26ba14e8ece4245b0d0c32ee880b564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } } }
I20260620 19:16:23.669131 7609 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:23.669064 7610 sys_catalog.cc:455] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d26ba14e8ece4245b0d0c32ee880b564" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d26ba14e8ece4245b0d0c32ee880b564" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38407 } } }
I20260620 19:16:23.669282 7610 sys_catalog.cc:458] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:23.669335 7615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:23.669811 7615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:23.671088 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:23.667785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56393 (local address 127.0.0.1:38407)
0620 19:16:23.667823 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:23.667825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:23.667923 (+ 98us) server_negotiation.cc:407] Connection header received
0620 19:16:23.667972 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:23.667973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:23.667983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:23.668039 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:23.668154 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:23.668721 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:23.669658 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:23.670002 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:23.670069 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:23.670729 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:23.670730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:23.670731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:23.670734 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:23.670748 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:23.670752 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:23.670813 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:23.671021 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 19:16:23.671028 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:16:23.671572 7615 catalog_manager.cc:1347] Generated new cluster ID: 77b5f5cc082f41c2918e5cde49fe8091
I20260620 19:16:23.671617 7615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:23.691066 7615 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:23.691591 7615 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:23.694993 7615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d26ba14e8ece4245b0d0c32ee880b564: Generated new TSK 0
I20260620 19:16:23.695154 7615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:23.703819 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7531
W20260620 19:16:23.707681 7516 connection.cc:535] client connection to 127.0.0.1:38407 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38407): Connection reset by peer (error 104)
2026-06-20T19:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1226 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:23 dist-test-slave-nq8r krb5kdc[7630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:23 dist-test-slave-nq8r krb5kdc[7630](info): set up 2 sockets
Jun 20 19:16:23 dist-test-slave-nq8r krb5kdc[7630](info): commencing operation
krb5kdc: starting...
W20260620 19:16:24.644572 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:24 dist-test-slave-nq8r krb5kdc[7630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:24.749743 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:24.827030 7643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:24.827219 7643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:24.827240 7643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:24.828370 7643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:24.828414 7643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:24.828428 7643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:24.828438 7643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:24.828449 7643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:24.829874 7643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:24.830083 7643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:24.830278 7643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:24.834561 7649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:24.834609 7651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:24.834656 7643 server_base.cc:1034] running on GCE node
W20260620 19:16:24.834748 7648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:24.835000 7643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:24.835270 7643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:24.836403 7643 hybrid_clock.cc:648] HybridClock initialized: now 1781982984836390 us; error 32 us; skew 500 ppm
Jun 20 19:16:24 dist-test-slave-nq8r krb5kdc[7630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:24.838788 7643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:24.839195 7643 webserver.cc:460] Webserver started at http://127.0.0.1:34229/ using document root <none> and password file <none>
I20260620 19:16:24.839397 7643 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:24.839443 7643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:24.839560 7643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:24.840519 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b"
format_stamp: "Formatted at 2026-06-20 19:16:24 on dist-test-slave-nq8r"
server_key: "b20cd6dcf29637f553339558dfbd924e"
server_key_iv: "e44b727ed2bdec780a40bfe47eb70caf"
server_key_version: "encryptionkey@0"
I20260620 19:16:24.840834 7643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b"
format_stamp: "Formatted at 2026-06-20 19:16:24 on dist-test-slave-nq8r"
server_key: "b20cd6dcf29637f553339558dfbd924e"
server_key_iv: "e44b727ed2bdec780a40bfe47eb70caf"
server_key_version: "encryptionkey@0"
I20260620 19:16:24.842015 7643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:24.842617 7658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:24.842773 7643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:24.842849 7643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b"
format_stamp: "Formatted at 2026-06-20 19:16:24 on dist-test-slave-nq8r"
server_key: "b20cd6dcf29637f553339558dfbd924e"
server_key_iv: "e44b727ed2bdec780a40bfe47eb70caf"
server_key_version: "encryptionkey@0"
I20260620 19:16:24.842905 7643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:24.846029 7643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:24.847606 7643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:24.847730 7643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:24.851271 7643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43421
I20260620 19:16:24.851300 7710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43421 every 8 connection(s)
I20260620 19:16:24.851578 7643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:24.852351 7711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:24.853535 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7643
I20260620 19:16:24.853638 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:24.853788 19723 external_mini_cluster.cc:1428] Setting key 9826fcf6d8bc1ddf7919bf72f597b864
I20260620 19:16:24.854733 7711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b: Bootstrap starting.
I20260620 19:16:24.855340 7711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:24.855607 7711 log.cc:824] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:24.855937 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:24.854231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33585 (local address 127.0.0.1:43421)
0620 19:16:24.854391 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:24.854399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:24.854422 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:16:24.854812 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:24.854821 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:24.854851 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:24.854960 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:24.855561 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:24.855565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:24.855565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:24.855569 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:24.855602 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:24.855611 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:24.855688 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:24.855719 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 19:16:24.855756 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
I20260620 19:16:24.856299 7711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b: No bootstrap required, opened a new log
I20260620 19:16:24.856803 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:24.858716 7711 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260620 19:16:24.858861 7711 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:24.858902 7711 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ef250bb0f04012a97a4ccb2f58ec3b, State: Initialized, Role: FOLLOWER
I20260620 19:16:24.859020 7711 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260620 19:16:24.859099 7711 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:24.859122 7711 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:24.859202 7711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:24.859213 7714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:24.858287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45737 (local address 127.0.0.1:43421)
0620 19:16:24.858324 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:24.858326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:24.858416 (+ 90us) server_negotiation.cc:407] Connection header received
0620 19:16:24.858476 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:24.858476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:24.858480 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:24.858529 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:24.859012 (+ 483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:24.859013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:24.859013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:24.859017 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:24.859028 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:24.859032 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:24.859072 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:24.859143 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 19:16:24.859155 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:16:24.859815 7711 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260620 19:16:24.859911 7711 leader_election.cc:302] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ef250bb0f04012a97a4ccb2f58ec3b; no voters:
I20260620 19:16:24.860059 7711 leader_election.cc:288] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:24.860095 7721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:24.860232 7721 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [term 1 LEADER]: Becoming Leader. State: Replica: b9ef250bb0f04012a97a4ccb2f58ec3b, State: Running, Role: LEADER
I20260620 19:16:24.860287 7711 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:24.860344 7721 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } }
I20260620 19:16:24.860800 7722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } } }
I20260620 19:16:24.860977 7723 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ef250bb0f04012a97a4ccb2f58ec3b. Latest consensus state: current_term: 1 leader_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef250bb0f04012a97a4ccb2f58ec3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43421 } } }
I20260620 19:16:24.861127 7723 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:24.861042 7722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:24.861342 7731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:24.861814 7731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:24.863263 7731 catalog_manager.cc:1347] Generated new cluster ID: 3fd368e449de4041a828c994bdc3fbfd
I20260620 19:16:24.863308 7731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:24.876545 7731 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:24.876729 7731 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:24.883423 7731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ef250bb0f04012a97a4ccb2f58ec3b: Generated new TSK 0
I20260620 19:16:24.883673 7731 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:24.889406 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7643
2026-06-20T19:16:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1185 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:24 dist-test-slave-nq8r krb5kdc[7742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:24 dist-test-slave-nq8r krb5kdc[7742](info): set up 2 sockets
Jun 20 19:16:24 dist-test-slave-nq8r krb5kdc[7742](info): commencing operation
krb5kdc: starting...
W20260620 19:16:25.840592 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:25 dist-test-slave-nq8r krb5kdc[7742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:25.947654 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:26.026186 7755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:26.026377 7755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:26.026398 7755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:26.027442 7755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:26.027474 7755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:26.027486 7755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:26.027498 7755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:26.027508 7755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:26.028899 7755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:26.029075 7755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:26.029237 7755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:26.033551 7761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:26.033634 7755 server_base.cc:1034] running on GCE node
W20260620 19:16:26.033556 7760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:26.033550 7763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:26.033928 7755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:26.034195 7755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:26.035344 7755 hybrid_clock.cc:648] HybridClock initialized: now 1781982986035331 us; error 28 us; skew 500 ppm
Jun 20 19:16:26 dist-test-slave-nq8r krb5kdc[7742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:26.037626 7755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:26.038028 7755 webserver.cc:460] Webserver started at http://127.0.0.1:40455/ using document root <none> and password file <none>
I20260620 19:16:26.038235 7755 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:26.038282 7755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:26.038404 7755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:26.039418 7755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "c6f588496d8b42609f74dd65b2f9ccad"
format_stamp: "Formatted at 2026-06-20 19:16:26 on dist-test-slave-nq8r"
server_key: "a926f787e61c8685e299cc322183090d"
server_key_iv: "20633739c95ad86db1089b2a0882debc"
server_key_version: "encryptionkey@0"
I20260620 19:16:26.039734 7755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "c6f588496d8b42609f74dd65b2f9ccad"
format_stamp: "Formatted at 2026-06-20 19:16:26 on dist-test-slave-nq8r"
server_key: "a926f787e61c8685e299cc322183090d"
server_key_iv: "20633739c95ad86db1089b2a0882debc"
server_key_version: "encryptionkey@0"
I20260620 19:16:26.040966 7755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:26.041661 7770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:26.041800 7755 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:26.041878 7755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "c6f588496d8b42609f74dd65b2f9ccad"
format_stamp: "Formatted at 2026-06-20 19:16:26 on dist-test-slave-nq8r"
server_key: "a926f787e61c8685e299cc322183090d"
server_key_iv: "20633739c95ad86db1089b2a0882debc"
server_key_version: "encryptionkey@0"
I20260620 19:16:26.041931 7755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:26.056588 7755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:26.058343 7755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:26.058454 7755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:26.062192 7755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40457
I20260620 19:16:26.062247 7822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40457 every 8 connection(s)
I20260620 19:16:26.062551 7755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:26.063369 7823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:26.065630 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad: Bootstrap starting.
I20260620 19:16:26.066156 7823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:26.066375 7823 log.cc:824] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:26.066992 7823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad: No bootstrap required, opened a new log
I20260620 19:16:26.068504 7823 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 19:16:26.068637 7823 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:26.068660 7823 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f588496d8b42609f74dd65b2f9ccad, State: Initialized, Role: FOLLOWER
I20260620 19:16:26.068768 7823 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 19:16:26.068815 7823 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:26.068840 7823 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:26.068909 7823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:26.069368 7823 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 19:16:26.069449 7823 leader_election.cc:302] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6f588496d8b42609f74dd65b2f9ccad; no voters:
I20260620 19:16:26.069562 7823 leader_election.cc:288] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:26.069603 7826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:26.069777 7823 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:26.069790 7826 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [term 1 LEADER]: Becoming Leader. State: Replica: c6f588496d8b42609f74dd65b2f9ccad, State: Running, Role: LEADER
I20260620 19:16:26.069893 7826 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } }
I20260620 19:16:26.070384 7827 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6f588496d8b42609f74dd65b2f9ccad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260620 19:16:26.070477 7827 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:26.070607 7828 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6f588496d8b42609f74dd65b2f9ccad. Latest consensus state: current_term: 1 leader_uuid: "c6f588496d8b42609f74dd65b2f9ccad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6f588496d8b42609f74dd65b2f9ccad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40457 } } }
I20260620 19:16:26.070731 7828 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:26.070751 7835 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:26.071177 7835 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:26.072113 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7755
I20260620 19:16:26.072216 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:26.072393 19723 external_mini_cluster.cc:1428] Setting key 830cddadcc36acafc8b3e6180ba92327
I20260620 19:16:26.072551 7835 catalog_manager.cc:1347] Generated new cluster ID: ef637fd856f04d4cadfe3056a5fafa4a
I20260620 19:16:26.072599 7835 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:16:26 dist-test-slave-nq8r krb5kdc[7742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:16:26.083439 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:26.072804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55413 (local address 127.0.0.1:40457)
0620 19:16:26.072962 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:26.072971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:26.073003 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:16:26.073125 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:26.073132 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:26.073185 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:26.073333 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:26.074032 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:26.075388 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:26.076457 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:26.077071 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:26.077166 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:26.079917 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:26.079923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:26.079924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:26.079936 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:26.081874 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:26.082236 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:26.082239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:26.082240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:26.082350 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:26.082516 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:26.082517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:26.082517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:26.082825 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:26.082992 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:26.083067 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 19:16:26.083201 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260620 19:16:26.083953 7835 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:26.084631 7835 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:26.085541 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:26.089766 7835 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6f588496d8b42609f74dd65b2f9ccad: Generated new TSK 0
I20260620 19:16:26.089928 7835 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:26.093669 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:26.086943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55977 (local address 127.0.0.1:40457)
0620 19:16:26.086980 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:26.086982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:26.087147 (+ 165us) server_negotiation.cc:407] Connection header received
0620 19:16:26.087242 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:26.087243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:26.087253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:26.087281 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:26.087929 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:26.088706 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:26.089827 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:26.090150 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:26.090186 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:26.091494 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:26.091495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:26.091495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:26.091501 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:26.092676 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:26.093078 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:26.093078 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:26.093079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:26.093123 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:26.093305 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:26.093305 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:26.093305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:26.093388 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:26.093453 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:26.093559 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 19:16:26.093614 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:26.095752 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7755
2026-06-20T19:16:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:26 dist-test-slave-nq8r krb5kdc[7854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:26 dist-test-slave-nq8r krb5kdc[7854](info): set up 2 sockets
Jun 20 19:16:26 dist-test-slave-nq8r krb5kdc[7854](info): commencing operation
krb5kdc: starting...
W20260620 19:16:27.040606 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:27 dist-test-slave-nq8r krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:27.145551 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:27.223318 7867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:27.223517 7867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:27.223547 7867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:27.224757 7867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:27.224802 7867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:27.224820 7867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:27.224835 7867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:27.224869 7867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:27.226323 7867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:27.226558 7867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:27.226761 7867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:27.231119 7875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:27.231086 7872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:27.231098 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:27.231235 7867 server_base.cc:1034] running on GCE node
I20260620 19:16:27.231495 7867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:27.231791 7867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:27.232900 7867 hybrid_clock.cc:648] HybridClock initialized: now 1781982987232886 us; error 24 us; skew 500 ppm
Jun 20 19:16:27 dist-test-slave-nq8r krb5kdc[7854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:27.235158 7867 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:27.235539 7867 webserver.cc:460] Webserver started at http://127.0.0.1:40023/ using document root <none> and password file <none>
I20260620 19:16:27.235750 7867 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:27.235797 7867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:27.235918 7867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:27.236950 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "d6e7d549407641cd914c0f94e382f342"
format_stamp: "Formatted at 2026-06-20 19:16:27 on dist-test-slave-nq8r"
server_key: "21c8dbe37cf408534df91a1045215969"
server_key_iv: "689a338c529694a354315f50a1d4441d"
server_key_version: "encryptionkey@0"
I20260620 19:16:27.237269 7867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "d6e7d549407641cd914c0f94e382f342"
format_stamp: "Formatted at 2026-06-20 19:16:27 on dist-test-slave-nq8r"
server_key: "21c8dbe37cf408534df91a1045215969"
server_key_iv: "689a338c529694a354315f50a1d4441d"
server_key_version: "encryptionkey@0"
I20260620 19:16:27.238428 7867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:16:27.239069 7882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:27.239233 7867 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:27.239323 7867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "d6e7d549407641cd914c0f94e382f342"
format_stamp: "Formatted at 2026-06-20 19:16:27 on dist-test-slave-nq8r"
server_key: "21c8dbe37cf408534df91a1045215969"
server_key_iv: "689a338c529694a354315f50a1d4441d"
server_key_version: "encryptionkey@0"
I20260620 19:16:27.239379 7867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:27.250473 7867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:27.252480 7867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:27.252583 7867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:27.256064 7867 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34083
I20260620 19:16:27.256098 7934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34083 every 8 connection(s)
I20260620 19:16:27.256421 7867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:27.257263 7935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:27.259466 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342: Bootstrap starting.
I20260620 19:16:27.259675 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7867
I20260620 19:16:27.259781 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:27.259974 19723 external_mini_cluster.cc:1428] Setting key 0be2f1c956de227967d3303a6f0b7343
I20260620 19:16:27.260157 7935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:27.260452 7935 log.cc:824] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:27.261098 7935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342: No bootstrap required, opened a new log
I20260620 19:16:27.263063 7935 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260620 19:16:27.263231 7935 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:27.263262 7935 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6e7d549407641cd914c0f94e382f342, State: Initialized, Role: FOLLOWER
I20260620 19:16:27.263381 7935 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260620 19:16:27.263451 7935 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:27.263515 7935 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:27.263566 7935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:27.264106 7935 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260620 19:16:27.264215 7935 leader_election.cc:302] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6e7d549407641cd914c0f94e382f342; no voters:
I20260620 19:16:27.264365 7935 leader_election.cc:288] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:27.264403 7940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:27.264566 7940 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [term 1 LEADER]: Becoming Leader. State: Replica: d6e7d549407641cd914c0f94e382f342, State: Running, Role: LEADER
I20260620 19:16:27.264593 7935 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:27.264719 7940 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } }
I20260620 19:16:27.265033 7940 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6e7d549407641cd914c0f94e382f342. Latest consensus state: current_term: 1 leader_uuid: "d6e7d549407641cd914c0f94e382f342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260620 19:16:27.265329 7940 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:27.265516 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:27.260443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47347 (local address 127.0.0.1:34083)
0620 19:16:27.260642 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:27.260650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:27.260674 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:16:27.261164 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:27.261173 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:27.261237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:27.261362 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:27.261559 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:27.262413 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:27.263484 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:27.263958 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:27.264037 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:27.264703 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:27.264707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:27.264707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:27.264926 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:27.264963 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:27.264977 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:27.265030 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:27.265252 (+ 222us) server_negotiation.cc:299] Negotiation successful
0620 19:16:27.265323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":72,"threads_started":1}
I20260620 19:16:27.265163 7941 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6e7d549407641cd914c0f94e382f342" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6e7d549407641cd914c0f94e382f342" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34083 } } }
I20260620 19:16:27.265681 7941 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:27.265815 7949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:27.266341 7949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:27.266544 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:27.267869 7949 catalog_manager.cc:1347] Generated new cluster ID: 24d7676f75774629a0698d3fca6439f7
I20260620 19:16:27.267915 7949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:27.271149 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:27.267884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36181 (local address 127.0.0.1:34083)
0620 19:16:27.267925 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:27.267927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:27.268154 (+ 227us) server_negotiation.cc:407] Connection header received
0620 19:16:27.268182 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:27.268182 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:27.268193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:27.268252 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:27.268342 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:27.269061 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:27.269995 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:27.270297 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:27.270364 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:27.270933 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:27.270934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:27.270935 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:27.270939 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:27.270953 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:27.270957 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:27.271012 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:27.271058 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 19:16:27.271078 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:27.285830 7949 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:27.286355 7949 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:27.295338 7949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6e7d549407641cd914c0f94e382f342: Generated new TSK 0
I20260620 19:16:27.295533 7949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:27.301970 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7867
2026-06-20T19:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:27 dist-test-slave-nq8r krb5kdc[7966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:27 dist-test-slave-nq8r krb5kdc[7966](info): set up 2 sockets
Jun 20 19:16:27 dist-test-slave-nq8r krb5kdc[7966](info): commencing operation
krb5kdc: starting...
W20260620 19:16:28.236507 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:28 dist-test-slave-nq8r krb5kdc[7966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:28.343125 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:28.420080 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:28.420300 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:28.420331 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:28.421499 7979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:28.421540 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:28.421558 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:28.421574 7979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:28.421588 7979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:28.423026 7979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:28.423276 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:28.423478 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:28.427908 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:28.427928 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:28.427932 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:28.428197 7979 server_base.cc:1034] running on GCE node
I20260620 19:16:28.428421 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:28.428681 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:28.429826 7979 hybrid_clock.cc:648] HybridClock initialized: now 1781982988429805 us; error 37 us; skew 500 ppm
Jun 20 19:16:28 dist-test-slave-nq8r krb5kdc[7966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:28.432070 7979 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:28.432467 7979 webserver.cc:460] Webserver started at http://127.0.0.1:37433/ using document root <none> and password file <none>
I20260620 19:16:28.432659 7979 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:28.432705 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:28.432829 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:28.434101 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "c445abdb587c4c2cb269e315ada3557d"
format_stamp: "Formatted at 2026-06-20 19:16:28 on dist-test-slave-nq8r"
server_key: "064677b25fb33c0b068a935a27d902c9"
server_key_iv: "6bd285f9abd32f2c67e4d7ee1839ad11"
server_key_version: "encryptionkey@0"
I20260620 19:16:28.434423 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "c445abdb587c4c2cb269e315ada3557d"
format_stamp: "Formatted at 2026-06-20 19:16:28 on dist-test-slave-nq8r"
server_key: "064677b25fb33c0b068a935a27d902c9"
server_key_iv: "6bd285f9abd32f2c67e4d7ee1839ad11"
server_key_version: "encryptionkey@0"
I20260620 19:16:28.435562 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:28.436159 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:28.436309 7979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:28.436386 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "c445abdb587c4c2cb269e315ada3557d"
format_stamp: "Formatted at 2026-06-20 19:16:28 on dist-test-slave-nq8r"
server_key: "064677b25fb33c0b068a935a27d902c9"
server_key_iv: "6bd285f9abd32f2c67e4d7ee1839ad11"
server_key_version: "encryptionkey@0"
I20260620 19:16:28.436450 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:28.439532 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:28.441468 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:28.441589 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:28.445003 7979 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34961
I20260620 19:16:28.445029 8046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34961 every 8 connection(s)
I20260620 19:16:28.445309 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:28.446048 8047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:28.446985 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 7979
I20260620 19:16:28.447077 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:28.447232 19723 external_mini_cluster.cc:1428] Setting key 2c6c5d98759916212ca0b9700df328e3
I20260620 19:16:28.448376 8047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d: Bootstrap starting.
I20260620 19:16:28.449048 8047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:28.449332 8047 log.cc:824] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:28.449399 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:28.447692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52879 (local address 127.0.0.1:34961)
0620 19:16:28.447849 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:28.447856 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:28.447886 (+ 30us) server_negotiation.cc:407] Connection header received
0620 19:16:28.448272 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:28.448278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:28.448301 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:28.448409 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:28.449001 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:28.449004 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:28.449004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:28.449008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:28.449043 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:28.449053 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:28.449129 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:28.449184 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 19:16:28.449218 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260620 19:16:28.449949 8047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d: No bootstrap required, opened a new log
I20260620 19:16:28.450212 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:28.451834 8047 raft_consensus.cc:348] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 19:16:28.451959 8047 raft_consensus.cc:374] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:28.451990 8047 raft_consensus.cc:729] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c445abdb587c4c2cb269e315ada3557d, State: Initialized, Role: FOLLOWER
I20260620 19:16:28.452103 8047 consensus_queue.cc:260] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 19:16:28.452178 8047 raft_consensus.cc:388] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:28.452208 8047 raft_consensus.cc:482] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:28.452253 8047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:28.452375 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:28.451570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38499 (local address 127.0.0.1:34961)
0620 19:16:28.451619 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:28.451621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:28.451628 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:28.451662 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:28.451663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:28.451667 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:28.451716 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:28.452173 (+ 457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:28.452174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:28.452175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:28.452179 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:28.452191 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:28.452194 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:28.452235 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:28.452307 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 19:16:28.452317 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:28.452940 8047 raft_consensus.cc:504] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 19:16:28.453028 8047 leader_election.cc:302] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c445abdb587c4c2cb269e315ada3557d; no voters:
I20260620 19:16:28.453167 8047 leader_election.cc:288] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:28.453203 8057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:28.453328 8057 raft_consensus.cc:686] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [term 1 LEADER]: Becoming Leader. State: Replica: c445abdb587c4c2cb269e315ada3557d, State: Running, Role: LEADER
I20260620 19:16:28.453378 8047 sys_catalog.cc:564] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:28.453410 8057 consensus_queue.cc:237] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } }
I20260620 19:16:28.453724 8057 sys_catalog.cc:455] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c445abdb587c4c2cb269e315ada3557d. Latest consensus state: current_term: 1 leader_uuid: "c445abdb587c4c2cb269e315ada3557d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260620 19:16:28.453753 8058 sys_catalog.cc:455] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c445abdb587c4c2cb269e315ada3557d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c445abdb587c4c2cb269e315ada3557d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34961 } } }
I20260620 19:16:28.453835 8057 sys_catalog.cc:458] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:28.453917 8058 sys_catalog.cc:458] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:28.454284 8065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:28.454655 8065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:28.456055 8065 catalog_manager.cc:1347] Generated new cluster ID: 3881e7cbfaef4d37b397082542894534
I20260620 19:16:28.456102 8065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:28.472996 8065 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:28.473186 8065 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:28.478003 8065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c445abdb587c4c2cb269e315ada3557d: Generated new TSK 0
I20260620 19:16:28.478165 8065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:28.480545 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 7979
2026-06-20T19:16:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1178 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:28 dist-test-slave-nq8r krb5kdc[8078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:28 dist-test-slave-nq8r krb5kdc[8078](info): set up 2 sockets
Jun 20 19:16:28 dist-test-slave-nq8r krb5kdc[8078](info): commencing operation
krb5kdc: starting...
W20260620 19:16:29.416618 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:29.521468 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:29.602140 8091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:29.602339 8091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:29.602370 8091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:29.603507 8091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:29.603551 8091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:29.603570 8091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:29.603585 8091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:29.603600 8091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:29.605072 8091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:29.605316 8091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:29.605515 8091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:29.609992 8097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:29.610056 8091 server_base.cc:1034] running on GCE node
W20260620 19:16:29.609982 8099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:29.609983 8096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:29.610423 8091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:29.610692 8091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:29.611832 8091 hybrid_clock.cc:648] HybridClock initialized: now 1781982989611819 us; error 30 us; skew 500 ppm
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:29.617365 8091 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:29.617805 8091 webserver.cc:460] Webserver started at http://127.0.0.1:38651/ using document root <none> and password file <none>
I20260620 19:16:29.618000 8091 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:29.618045 8091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:29.618167 8091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:29.619203 8091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "41988ff22d9245fdbfb28ba243f75c4e"
format_stamp: "Formatted at 2026-06-20 19:16:29 on dist-test-slave-nq8r"
server_key: "5d17720a13e742b87fc58ffe8c3125db"
server_key_iv: "2b81cb0d03a8cbeca8957af6aade96fc"
server_key_version: "encryptionkey@0"
I20260620 19:16:29.619532 8091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "41988ff22d9245fdbfb28ba243f75c4e"
format_stamp: "Formatted at 2026-06-20 19:16:29 on dist-test-slave-nq8r"
server_key: "5d17720a13e742b87fc58ffe8c3125db"
server_key_iv: "2b81cb0d03a8cbeca8957af6aade96fc"
server_key_version: "encryptionkey@0"
I20260620 19:16:29.620697 8091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:29.621408 8106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:29.621551 8091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:29.621632 8091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "41988ff22d9245fdbfb28ba243f75c4e"
format_stamp: "Formatted at 2026-06-20 19:16:29 on dist-test-slave-nq8r"
server_key: "5d17720a13e742b87fc58ffe8c3125db"
server_key_iv: "2b81cb0d03a8cbeca8957af6aade96fc"
server_key_version: "encryptionkey@0"
I20260620 19:16:29.621697 8091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:29.629894 8091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:29.631937 8091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:29.632061 8091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:29.635767 8091 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42801
I20260620 19:16:29.635798 8158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42801 every 8 connection(s)
I20260620 19:16:29.636125 8091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:29.636968 8159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:29.639148 8159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e: Bootstrap starting.
I20260620 19:16:29.639655 8159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:29.639885 8159 log.cc:824] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:29.640501 8159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e: No bootstrap required, opened a new log
I20260620 19:16:29.642063 8159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 19:16:29.642199 8159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:29.642221 8159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41988ff22d9245fdbfb28ba243f75c4e, State: Initialized, Role: FOLLOWER
I20260620 19:16:29.642329 8159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 19:16:29.642400 8159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:29.642422 8159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:29.642454 8159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:29.642920 8159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 19:16:29.643004 8159 leader_election.cc:302] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41988ff22d9245fdbfb28ba243f75c4e; no voters:
I20260620 19:16:29.643126 8159 leader_election.cc:288] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:29.643172 8162 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:29.643333 8162 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [term 1 LEADER]: Becoming Leader. State: Replica: 41988ff22d9245fdbfb28ba243f75c4e, State: Running, Role: LEADER
I20260620 19:16:29.643344 8159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:29.643433 8162 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } }
I20260620 19:16:29.643813 8163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41988ff22d9245fdbfb28ba243f75c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } } }
I20260620 19:16:29.643937 8163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:29.643930 8164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41988ff22d9245fdbfb28ba243f75c4e. Latest consensus state: current_term: 1 leader_uuid: "41988ff22d9245fdbfb28ba243f75c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41988ff22d9245fdbfb28ba243f75c4e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42801 } } }
I20260620 19:16:29.644066 8164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:29.644464 8171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:29.644919 8171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:29.645778 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8091
I20260620 19:16:29.645881 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:29.646034 19723 external_mini_cluster.cc:1428] Setting key 773d582039cd689255efa5d4a61b0ff1
I20260620 19:16:29.646788 8171 catalog_manager.cc:1347] Generated new cluster ID: a253cc2e74b941fca4f4956d6f781695
I20260620 19:16:29.646836 8171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:16:29.657210 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:29.646605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:34687 (local address 127.0.0.1:42801)
0620 19:16:29.646866 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:29.646874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:29.646905 (+ 31us) server_negotiation.cc:407] Connection header received
0620 19:16:29.647012 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:29.647019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:29.647072 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:29.647196 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:29.647915 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:29.649287 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:29.650334 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:29.650933 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:29.651028 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:29.653653 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:29.653657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:29.653658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:29.653670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:29.655539 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:29.655937 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:29.655940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:29.655941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:29.656045 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:29.656230 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:29.656231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:29.656231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:29.656545 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:29.656717 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:29.656834 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 19:16:29.656970 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":69,"threads_started":1}
I20260620 19:16:29.658120 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:29.664899 8171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:29.665486 8171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:29.671617 8182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:29.661085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:51335 (local address 127.0.0.1:42801)
0620 19:16:29.661124 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:29.661126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:29.664515 (+ 3389us) server_negotiation.cc:407] Connection header received
0620 19:16:29.664543 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:29.664544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:29.664554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:29.664599 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:29.665283 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:29.666323 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:29.667289 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:29.667611 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:29.667677 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:29.669027 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:29.669028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:29.669028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:29.669035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:29.670516 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:29.670893 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:29.670894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:29.670894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:29.670953 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:29.671135 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:29.671136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:29.671136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:29.671271 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:29.671370 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:29.671466 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:16:29.671527 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:16:29.673005 8171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41988ff22d9245fdbfb28ba243f75c4e: Generated new TSK 0
I20260620 19:16:29.673178 8171 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:29.674782 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8091
W20260620 19:16:29.678725 8076 connection.cc:535] client connection to 127.0.0.1:42801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42801): Connection reset by peer (error 104)
2026-06-20T19:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8190](info): set up 2 sockets
Jun 20 19:16:29 dist-test-slave-nq8r krb5kdc[8190](info): commencing operation
krb5kdc: starting...
W20260620 19:16:30.616644 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:30 dist-test-slave-nq8r krb5kdc[8190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:30.721335 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34055
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:30.799685 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:30.799906 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:30.799928 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:30.801096 8203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:30.801139 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:30.801151 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:30.801164 8203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:30.801196 8203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:30.802655 8203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34055
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34055
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:30.802877 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:30.803081 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:30.807727 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:30.807737 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:30.807747 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:30.807770 8203 server_base.cc:1034] running on GCE node
I20260620 19:16:30.808094 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:30.808405 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:30.809545 8203 hybrid_clock.cc:648] HybridClock initialized: now 1781982990809533 us; error 24 us; skew 500 ppm
Jun 20 19:16:30 dist-test-slave-nq8r krb5kdc[8190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:30.811857 8203 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:30.812288 8203 webserver.cc:460] Webserver started at http://127.0.0.1:40081/ using document root <none> and password file <none>
I20260620 19:16:30.812479 8203 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:30.812525 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:30.812646 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:30.813727 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "759310868c564630abf701227ec402ca"
format_stamp: "Formatted at 2026-06-20 19:16:30 on dist-test-slave-nq8r"
server_key: "095d72b81e15732f5d741e491908cffc"
server_key_iv: "9b292a8adac8ba2917455c4fe48b9cab"
server_key_version: "encryptionkey@0"
I20260620 19:16:30.814071 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "759310868c564630abf701227ec402ca"
format_stamp: "Formatted at 2026-06-20 19:16:30 on dist-test-slave-nq8r"
server_key: "095d72b81e15732f5d741e491908cffc"
server_key_iv: "9b292a8adac8ba2917455c4fe48b9cab"
server_key_version: "encryptionkey@0"
I20260620 19:16:30.815305 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:30.815939 8218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:30.816090 8203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:30.816177 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "759310868c564630abf701227ec402ca"
format_stamp: "Formatted at 2026-06-20 19:16:30 on dist-test-slave-nq8r"
server_key: "095d72b81e15732f5d741e491908cffc"
server_key_iv: "9b292a8adac8ba2917455c4fe48b9cab"
server_key_version: "encryptionkey@0"
I20260620 19:16:30.816246 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:30.844249 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:30.846542 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:30.846680 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:30.850461 8203 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34055
I20260620 19:16:30.850505 8270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34055 every 8 connection(s)
I20260620 19:16:30.850831 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:30.851643 8271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:30.853880 8271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca: Bootstrap starting.
I20260620 19:16:30.854403 8271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:30.854620 8271 log.cc:824] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:30.855223 8271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca: No bootstrap required, opened a new log
I20260620 19:16:30.855768 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8203
I20260620 19:16:30.855870 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:30.856031 19723 external_mini_cluster.cc:1428] Setting key 23775892343f5905775e34633322e5d6
I20260620 19:16:30.857179 8271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 19:16:30.857357 8271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:30.857434 8271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 759310868c564630abf701227ec402ca, State: Initialized, Role: FOLLOWER
I20260620 19:16:30.857568 8271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 19:16:30.857651 8271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:30.857683 8271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:30.857726 8271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:30.858415 8271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 19:16:30.858521 8271 leader_election.cc:302] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 759310868c564630abf701227ec402ca; no voters:
I20260620 19:16:30.858678 8271 leader_election.cc:288] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:30.858717 8276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:30.858845 8276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [term 1 LEADER]: Becoming Leader. State: Replica: 759310868c564630abf701227ec402ca, State: Running, Role: LEADER
I20260620 19:16:30.858904 8271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:30.858979 8276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } }
I20260620 19:16:30.859301 8276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 759310868c564630abf701227ec402ca. Latest consensus state: current_term: 1 leader_uuid: "759310868c564630abf701227ec402ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260620 19:16:30.859417 8276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:30.859654 8285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:30.860140 8278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "759310868c564630abf701227ec402ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "759310868c564630abf701227ec402ca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34055 } } }
I20260620 19:16:30.860214 8278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:30.860369 8285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:30.862150 8285 catalog_manager.cc:1347] Generated new cluster ID: 40f413549d3741ac94ef072b69a03ffd
I20260620 19:16:30.862195 8285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:30.862432 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:30.856502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:56163 (local address 127.0.0.1:34055)
0620 19:16:30.856697 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:30.856707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:30.856743 (+ 36us) server_negotiation.cc:407] Connection header received
0620 19:16:30.857143 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:30.857151 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:30.857219 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:30.857366 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:30.857524 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:30.859168 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:30.860207 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:30.860789 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:30.860906 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:30.861704 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:30.861708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:30.861708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:30.861907 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:30.861944 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:30.861959 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:30.862033 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:30.862208 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:16:30.862236 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260620 19:16:30.863339 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:30.868245 8275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:30.864625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:55397 (local address 127.0.0.1:34055)
0620 19:16:30.864655 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:30.864656 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:30.864913 (+ 257us) server_negotiation.cc:407] Connection header received
0620 19:16:30.865012 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:30.865013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:30.865024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:30.865051 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:30.865244 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:30.865944 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:30.866948 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:30.867266 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:30.867320 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:30.868009 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:30.868010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:30.868010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:30.868014 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:30.868028 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:30.868032 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:30.868079 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:30.868155 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 19:16:30.868166 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:30.878995 8285 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:30.879711 8285 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:30.887274 8285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 759310868c564630abf701227ec402ca: Generated new TSK 0
I20260620 19:16:30.887491 8285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:30.894528 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8203
W20260620 19:16:30.898250 8188 connection.cc:535] client connection to 127.0.0.1:34055 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34055): Connection reset by peer (error 104)
2026-06-20T19:16:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:30 dist-test-slave-nq8r krb5kdc[8302](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:30 dist-test-slave-nq8r krb5kdc[8302](info): set up 2 sockets
Jun 20 19:16:30 dist-test-slave-nq8r krb5kdc[8302](info): commencing operation
krb5kdc: starting...
W20260620 19:16:31.832593 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:31 dist-test-slave-nq8r krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:31.937613 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:32.015131 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:32.015322 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:32.015352 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:32.016464 8315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:32.016503 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:32.016522 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:32.016538 8315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:32.016553 8315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:32.018020 8315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:32.018246 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:32.018438 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:32.022761 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:32.022783 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:32.022763 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:32.022883 8315 server_base.cc:1034] running on GCE node
I20260620 19:16:32.023123 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:32.023375 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:32.024506 8315 hybrid_clock.cc:648] HybridClock initialized: now 1781982992024492 us; error 28 us; skew 500 ppm
Jun 20 19:16:32 dist-test-slave-nq8r krb5kdc[8302](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:32.026737 8315 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:32.027120 8315 webserver.cc:460] Webserver started at http://127.0.0.1:46255/ using document root <none> and password file <none>
I20260620 19:16:32.027335 8315 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:32.027383 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:32.027503 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:32.028486 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "182611974ba24d028734cae72e1f4070"
format_stamp: "Formatted at 2026-06-20 19:16:32 on dist-test-slave-nq8r"
server_key: "259798d270af77e4280087bc5dfea425"
server_key_iv: "3fe2de0e1a68aa3c032c9ad6fdd1e81b"
server_key_version: "encryptionkey@0"
I20260620 19:16:32.028795 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "182611974ba24d028734cae72e1f4070"
format_stamp: "Formatted at 2026-06-20 19:16:32 on dist-test-slave-nq8r"
server_key: "259798d270af77e4280087bc5dfea425"
server_key_iv: "3fe2de0e1a68aa3c032c9ad6fdd1e81b"
server_key_version: "encryptionkey@0"
I20260620 19:16:32.029979 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:32.030586 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:32.030730 8315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:32.030807 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "182611974ba24d028734cae72e1f4070"
format_stamp: "Formatted at 2026-06-20 19:16:32 on dist-test-slave-nq8r"
server_key: "259798d270af77e4280087bc5dfea425"
server_key_iv: "3fe2de0e1a68aa3c032c9ad6fdd1e81b"
server_key_version: "encryptionkey@0"
I20260620 19:16:32.030869 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:32.034087 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:32.035678 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:32.035799 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:32.039274 8315 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42987
I20260620 19:16:32.039304 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42987 every 8 connection(s)
I20260620 19:16:32.039582 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:32.040350 8383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:32.041411 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8315
I20260620 19:16:32.041517 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:32.041692 19723 external_mini_cluster.cc:1428] Setting key 0fbdb2f85a855dce022aad9677d48e0f
I20260620 19:16:32.042906 8383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: Bootstrap starting.
I20260620 19:16:32.043540 8383 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:32.043828 8383 log.cc:824] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:32.043890 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:32.042159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:58745 (local address 127.0.0.1:42987)
0620 19:16:32.042327 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:32.042335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:32.042358 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:16:32.042747 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:32.042753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:32.042779 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:32.042891 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:32.043474 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:32.043477 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:32.043477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:32.043480 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:32.043520 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:32.043529 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:32.043597 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:32.043671 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 19:16:32.043710 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260620 19:16:32.044669 8383 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: No bootstrap required, opened a new log
I20260620 19:16:32.044812 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:32.047199 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:32.046322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:34641 (local address 127.0.0.1:42987)
0620 19:16:32.046357 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:32.046359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:32.046383 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:16:32.046448 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:32.046449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:32.046452 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:32.046492 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:32.046978 (+ 486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:32.046979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:32.046979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:32.046983 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:32.046993 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:32.046997 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:32.047029 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:32.047080 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 19:16:32.047089 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:32.047086 8383 raft_consensus.cc:348] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260620 19:16:32.047245 8383 raft_consensus.cc:374] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:32.047277 8383 raft_consensus.cc:729] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182611974ba24d028734cae72e1f4070, State: Initialized, Role: FOLLOWER
I20260620 19:16:32.047387 8383 consensus_queue.cc:260] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260620 19:16:32.047477 8383 raft_consensus.cc:388] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:32.047514 8383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:32.047562 8383 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:32.048086 8383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260620 19:16:32.048174 8383 leader_election.cc:302] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 182611974ba24d028734cae72e1f4070; no voters:
I20260620 19:16:32.048322 8383 leader_election.cc:288] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:32.048362 8393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:32.048529 8383 sys_catalog.cc:564] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:32.048534 8393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [term 1 LEADER]: Becoming Leader. State: Replica: 182611974ba24d028734cae72e1f4070, State: Running, Role: LEADER
I20260620 19:16:32.048717 8393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } }
I20260620 19:16:32.049072 8394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "182611974ba24d028734cae72e1f4070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } } }
I20260620 19:16:32.049153 8394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:32.049329 8395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 182611974ba24d028734cae72e1f4070. Latest consensus state: current_term: 1 leader_uuid: "182611974ba24d028734cae72e1f4070" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "182611974ba24d028734cae72e1f4070" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42987 } } }
I20260620 19:16:32.049386 8395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070 [sys.catalog]: This master's current role is: LEADER
W20260620 19:16:32.050240 8409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:32.050287 8409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:32.050340 8402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:32.050518 8402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:32.051683 8402 catalog_manager.cc:1347] Generated new cluster ID: 740cee7f32724351a3739bef70be1a90
I20260620 19:16:32.051731 8402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:32.061277 8402 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:32.061452 8402 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:32.069010 8402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 182611974ba24d028734cae72e1f4070: Generated new TSK 0
I20260620 19:16:32.069226 8402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:32.071707 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8315
2026-06-20T19:16:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1177 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:32 dist-test-slave-nq8r krb5kdc[8414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:32 dist-test-slave-nq8r krb5kdc[8414](info): set up 2 sockets
Jun 20 19:16:32 dist-test-slave-nq8r krb5kdc[8414](info): commencing operation
krb5kdc: starting...
W20260620 19:16:33.008662 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:33.114980 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:33.192693 8427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:33.192924 8427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:33.192948 8427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:33.194085 8427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:33.194124 8427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:33.194136 8427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:33.194159 8427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:33.194182 8427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:33.195608 8427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:33.195833 8427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:33.196034 8427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:33.200493 8433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:33.200502 8432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:33.200589 8427 server_base.cc:1034] running on GCE node
W20260620 19:16:33.200482 8435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:33.200984 8427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:33.201256 8427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:33.202400 8427 hybrid_clock.cc:648] HybridClock initialized: now 1781982993202386 us; error 28 us; skew 500 ppm
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:33.204619 8427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:33.205066 8427 webserver.cc:460] Webserver started at http://127.0.0.1:36515/ using document root <none> and password file <none>
I20260620 19:16:33.205276 8427 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:33.205322 8427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:33.205443 8427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:33.206425 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "3141c372e6d948a5b579cc29add10385"
format_stamp: "Formatted at 2026-06-20 19:16:33 on dist-test-slave-nq8r"
server_key: "3401e88c1d2fd43f1ea7ed68def09fc9"
server_key_iv: "06738215e5fd3aad3c7a0ef996f5edbe"
server_key_version: "encryptionkey@0"
I20260620 19:16:33.206763 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "3141c372e6d948a5b579cc29add10385"
format_stamp: "Formatted at 2026-06-20 19:16:33 on dist-test-slave-nq8r"
server_key: "3401e88c1d2fd43f1ea7ed68def09fc9"
server_key_iv: "06738215e5fd3aad3c7a0ef996f5edbe"
server_key_version: "encryptionkey@0"
I20260620 19:16:33.207928 8427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:33.208534 8442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:33.208698 8427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:33.208781 8427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "3141c372e6d948a5b579cc29add10385"
format_stamp: "Formatted at 2026-06-20 19:16:33 on dist-test-slave-nq8r"
server_key: "3401e88c1d2fd43f1ea7ed68def09fc9"
server_key_iv: "06738215e5fd3aad3c7a0ef996f5edbe"
server_key_version: "encryptionkey@0"
I20260620 19:16:33.208873 8427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:33.221599 8427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:33.223280 8427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:33.223402 8427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:33.226945 8427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41335
I20260620 19:16:33.226999 8494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41335 every 8 connection(s)
I20260620 19:16:33.227301 8427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:33.228081 8495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:33.229028 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8427
I20260620 19:16:33.229133 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:33.229311 19723 external_mini_cluster.cc:1428] Setting key 1e2bc2a63705fe15348dc742f4dab5e3
I20260620 19:16:33.230626 8495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385: Bootstrap starting.
I20260620 19:16:33.231315 8495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:33.231593 8495 log.cc:824] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:33.232191 8495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385: No bootstrap required, opened a new log
I20260620 19:16:33.234473 8495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260620 19:16:33.234644 8495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:33.234679 8495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3141c372e6d948a5b579cc29add10385, State: Initialized, Role: FOLLOWER
I20260620 19:16:33.234813 8495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260620 19:16:33.234896 8495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:33.234933 8495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:33.234982 8495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:33.235680 8495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260620 19:16:33.235788 8495 leader_election.cc:302] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3141c372e6d948a5b579cc29add10385; no voters:
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 19:16:33.235942 8495 leader_election.cc:288] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:33.235975 8500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:33.236119 8500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [term 1 LEADER]: Becoming Leader. State: Replica: 3141c372e6d948a5b579cc29add10385, State: Running, Role: LEADER
I20260620 19:16:33.236191 8495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:33.236246 8500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } }
I20260620 19:16:33.236609 8500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3141c372e6d948a5b579cc29add10385. Latest consensus state: current_term: 1 leader_uuid: "3141c372e6d948a5b579cc29add10385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } } }
I20260620 19:16:33.236680 8502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3141c372e6d948a5b579cc29add10385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3141c372e6d948a5b579cc29add10385" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41335 } } }
I20260620 19:16:33.236994 8500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:33.237005 8502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:33.237619 8509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:33.238067 8509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:33.240010 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:33.229809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:46875 (local address 127.0.0.1:41335)
0620 19:16:33.229954 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:33.229963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:33.229988 (+ 25us) server_negotiation.cc:407] Connection header received
0620 19:16:33.230470 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:33.230477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:33.230525 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:33.230635 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:33.231371 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:33.232306 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:33.233324 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:33.233722 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:33.233796 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:33.236419 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:33.236424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:33.236424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:33.236435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:33.238343 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:33.238766 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:33.238769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:33.238770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:33.238872 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:33.239116 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:33.239117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:33.239117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:33.239398 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:33.239556 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:33.239704 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 19:16:33.239811 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260620 19:16:33.240509 8509 catalog_manager.cc:1347] Generated new cluster ID: dac522a67a76442495b0d1d49e446e39
I20260620 19:16:33.240581 8509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:33.240948 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:33.249583 8498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:33.242357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:53849 (local address 127.0.0.1:41335)
0620 19:16:33.242392 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:33.242393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:33.242502 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:16:33.242590 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:33.242591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:33.242600 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:33.242627 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:33.243302 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:33.244065 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:33.245108 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:33.245415 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:33.245455 (+ 40us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:33.246889 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:33.246891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:33.246891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:33.246898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:33.248292 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:33.248684 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:33.248685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:33.248686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:33.248749 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:33.249008 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:33.249009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:33.249009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:33.249129 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:33.249204 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:33.249425 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 19:16:33.249488 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:16:33.250946 8509 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:33.251421 8509 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:33.258927 8509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3141c372e6d948a5b579cc29add10385: Generated new TSK 0
I20260620 19:16:33.259075 8509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:33.262363 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8427
W20260620 19:16:33.266182 8412 connection.cc:535] client connection to 127.0.0.1:41335 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41335): Connection reset by peer (error 104)
2026-06-20T19:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8526](info): set up 2 sockets
Jun 20 19:16:33 dist-test-slave-nq8r krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260620 19:16:34.204610 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:34 dist-test-slave-nq8r krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:34.308964 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:34.387588 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:34.387796 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:34.387817 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:34.388953 8539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:34.388990 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:34.389003 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:34.389015 8539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:34.389038 8539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:34.390451 8539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:34.390676 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:34.390877 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:34.395202 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:34.395219 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:34.395308 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:34.395423 8539 server_base.cc:1034] running on GCE node
I20260620 19:16:34.395625 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:34.395854 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:34.396983 8539 hybrid_clock.cc:648] HybridClock initialized: now 1781982994396974 us; error 26 us; skew 500 ppm
Jun 20 19:16:34 dist-test-slave-nq8r krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:34.399242 8539 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:34.399632 8539 webserver.cc:460] Webserver started at http://127.0.0.1:37657/ using document root <none> and password file <none>
I20260620 19:16:34.399804 8539 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:34.399842 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:34.399933 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:34.400931 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "727d6fb023dc45b884ae60a8bb072ad1"
format_stamp: "Formatted at 2026-06-20 19:16:34 on dist-test-slave-nq8r"
server_key: "90b5352d3f455376e9a66eed4be216a2"
server_key_iv: "6ec428ba51051e3e2c48f2394cfd6be9"
server_key_version: "encryptionkey@0"
I20260620 19:16:34.401228 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "727d6fb023dc45b884ae60a8bb072ad1"
format_stamp: "Formatted at 2026-06-20 19:16:34 on dist-test-slave-nq8r"
server_key: "90b5352d3f455376e9a66eed4be216a2"
server_key_iv: "6ec428ba51051e3e2c48f2394cfd6be9"
server_key_version: "encryptionkey@0"
I20260620 19:16:34.402339 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 19:16:34.402935 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:34.403079 8539 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:34.403157 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "727d6fb023dc45b884ae60a8bb072ad1"
format_stamp: "Formatted at 2026-06-20 19:16:34 on dist-test-slave-nq8r"
server_key: "90b5352d3f455376e9a66eed4be216a2"
server_key_iv: "6ec428ba51051e3e2c48f2394cfd6be9"
server_key_version: "encryptionkey@0"
I20260620 19:16:34.403213 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:34.413928 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:34.415644 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:34.415786 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:34.419557 8539 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37489
I20260620 19:16:34.419595 8606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37489 every 8 connection(s)
I20260620 19:16:34.419922 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:34.420722 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:34.422955 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8539
I20260620 19:16:34.423069 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:34.423080 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: Bootstrap starting.
I20260620 19:16:34.423251 19723 external_mini_cluster.cc:1428] Setting key ba9f1f07156f795cc38c44c761c83c88
I20260620 19:16:34.423733 8607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:34.424033 8607 log.cc:824] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:34.424705 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: No bootstrap required, opened a new log
I20260620 19:16:34.426712 8607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } }
I20260620 19:16:34.426899 8607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:34.426949 8607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727d6fb023dc45b884ae60a8bb072ad1, State: Initialized, Role: FOLLOWER
I20260620 19:16:34.427080 8607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } }
I20260620 19:16:34.427152 8607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:34.427183 8607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:34.427229 8607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:34.427896 8607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } }
I20260620 19:16:34.428002 8607 leader_election.cc:302] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 727d6fb023dc45b884ae60a8bb072ad1; no voters:
I20260620 19:16:34.428165 8607 leader_election.cc:288] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:34.428208 8612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:34.428355 8612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 727d6fb023dc45b884ae60a8bb072ad1, State: Running, Role: LEADER
I20260620 19:16:34.428417 8607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:34.428539 8612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } }
I20260620 19:16:34.428921 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "727d6fb023dc45b884ae60a8bb072ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } } }
I20260620 19:16:34.429004 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:34.429121 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:34.423746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:58837 (local address 127.0.0.1:37489)
0620 19:16:34.423910 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:34.423918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:34.423944 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:16:34.424466 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:34.424476 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:34.424531 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:34.424648 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:34.424896 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:34.425772 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:34.426854 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:34.427418 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:34.427495 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:34.428190 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:34.428194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:34.428194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:34.428419 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:34.428460 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:34.428476 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:34.428553 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:34.428760 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:16:34.428807 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260620 19:16:34.428913 8612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 727d6fb023dc45b884ae60a8bb072ad1. Latest consensus state: current_term: 1 leader_uuid: "727d6fb023dc45b884ae60a8bb072ad1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "727d6fb023dc45b884ae60a8bb072ad1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37489 } } }
I20260620 19:16:34.429270 8612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:34.430053 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 19:16:34.430125 8627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:34.430181 8627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:34.430264 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:34.430428 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:34.431914 8621 catalog_manager.cc:1347] Generated new cluster ID: eca778fa97024706911158ce8064b525
I20260620 19:16:34.431963 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:34.434971 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:34.431505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:48747 (local address 127.0.0.1:37489)
0620 19:16:34.431543 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:34.431545 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:34.431654 (+ 109us) server_negotiation.cc:407] Connection header received
0620 19:16:34.431705 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:34.431706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:34.431715 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:34.431754 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:34.431912 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:34.432633 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:34.433666 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:34.434002 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:34.434075 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:34.434734 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:34.434735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:34.434735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:34.434740 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:34.434756 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:34.434761 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:34.434818 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:34.434885 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 19:16:34.434896 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:16:34.444541 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:34.445124 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:34.455740 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 727d6fb023dc45b884ae60a8bb072ad1: Generated new TSK 0
I20260620 19:16:34.455938 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:34.460685 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8539
W20260620 19:16:34.464416 8524 connection.cc:535] client connection to 127.0.0.1:37489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37489): Connection reset by peer (error 104)
2026-06-20T19:16:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:34 dist-test-slave-nq8r krb5kdc[8638](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:34 dist-test-slave-nq8r krb5kdc[8638](info): set up 2 sockets
Jun 20 19:16:34 dist-test-slave-nq8r krb5kdc[8638](info): commencing operation
krb5kdc: starting...
W20260620 19:16:35.400516 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:35 dist-test-slave-nq8r krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 19:16:35.508680 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:35.590467 8651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:35.590670 8651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:35.590691 8651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:35.591830 8651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:35.591872 8651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:35.591885 8651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:35.591907 8651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:35.591930 8651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:35.593437 8651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34189
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:35.593669 8651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:35.593878 8651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:35.598361 8657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:35.598353 8656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:35.598466 8651 server_base.cc:1034] running on GCE node
W20260620 19:16:35.598356 8659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:35.598773 8651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:35.599079 8651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:35.600231 8651 hybrid_clock.cc:648] HybridClock initialized: now 1781982995600215 us; error 27 us; skew 500 ppm
Jun 20 19:16:35 dist-test-slave-nq8r krb5kdc[8638](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:35.605624 8651 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 19:16:35.606047 8651 webserver.cc:460] Webserver started at http://127.0.0.1:33701/ using document root <none> and password file <none>
I20260620 19:16:35.606244 8651 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:35.606292 8651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:35.606411 8651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:35.607435 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "58a95636f50a4f4f8296c1c48b07c83e"
format_stamp: "Formatted at 2026-06-20 19:16:35 on dist-test-slave-nq8r"
server_key: "743ac43f8a6037ae4b4914740a90b860"
server_key_iv: "ec680da22250ad6c97701cd6dc7690d7"
server_key_version: "encryptionkey@0"
I20260620 19:16:35.607748 8651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "58a95636f50a4f4f8296c1c48b07c83e"
format_stamp: "Formatted at 2026-06-20 19:16:35 on dist-test-slave-nq8r"
server_key: "743ac43f8a6037ae4b4914740a90b860"
server_key_iv: "ec680da22250ad6c97701cd6dc7690d7"
server_key_version: "encryptionkey@0"
I20260620 19:16:35.608973 8651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:35.609609 8666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:35.609735 8651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:35.609822 8651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "58a95636f50a4f4f8296c1c48b07c83e"
format_stamp: "Formatted at 2026-06-20 19:16:35 on dist-test-slave-nq8r"
server_key: "743ac43f8a6037ae4b4914740a90b860"
server_key_iv: "ec680da22250ad6c97701cd6dc7690d7"
server_key_version: "encryptionkey@0"
I20260620 19:16:35.609885 8651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:35.613040 8651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:35.615020 8651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:35.615139 8651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:35.618705 8651 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34189
I20260620 19:16:35.618734 8718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34189 every 8 connection(s)
I20260620 19:16:35.619038 8651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:35.619831 8719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:35.622016 8719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e: Bootstrap starting.
I20260620 19:16:35.622530 8719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:35.622784 8719 log.cc:824] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:35.622934 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8651
I20260620 19:16:35.623041 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:35.623181 19723 external_mini_cluster.cc:1428] Setting key 5e10ee15a04a1d8461633e5e20ba924a
I20260620 19:16:35.623425 8719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e: No bootstrap required, opened a new log
I20260620 19:16:35.625442 8719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260620 19:16:35.625610 8719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:35.625701 8719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a95636f50a4f4f8296c1c48b07c83e, State: Initialized, Role: FOLLOWER
I20260620 19:16:35.625842 8719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260620 19:16:35.625933 8719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:35.625967 8719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:35.626004 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:35.623646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:46455 (local address 127.0.0.1:34189)
0620 19:16:35.623834 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:35.623844 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:35.623876 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:16:35.624613 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:35.624623 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:35.624657 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:35.624815 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:35.625385 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:35.625388 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:35.625388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:35.625393 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:35.625440 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:35.625453 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:35.625523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:35.625562 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 19:16:35.625603 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260620 19:16:35.626014 8719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:35.626619 8719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260620 19:16:35.626710 8719 leader_election.cc:302] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58a95636f50a4f4f8296c1c48b07c83e; no voters:
I20260620 19:16:35.626856 8719 leader_election.cc:288] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:35.626865 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:35.626885 8724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:35.627025 8724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [term 1 LEADER]: Becoming Leader. State: Replica: 58a95636f50a4f4f8296c1c48b07c83e, State: Running, Role: LEADER
I20260620 19:16:35.627089 8719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:35.627156 8724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } }
I20260620 19:16:35.627478 8724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58a95636f50a4f4f8296c1c48b07c83e. Latest consensus state: current_term: 1 leader_uuid: "58a95636f50a4f4f8296c1c48b07c83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260620 19:16:35.627691 8724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:35.627620 8726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58a95636f50a4f4f8296c1c48b07c83e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58a95636f50a4f4f8296c1c48b07c83e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34189 } } }
I20260620 19:16:35.628072 8726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:35.628245 8733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:35.628703 8733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:35.629415 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:35.628461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:50669 (local address 127.0.0.1:34189)
0620 19:16:35.628499 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:35.628501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:35.628508 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:35.628528 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:35.628529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:35.628533 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:35.628577 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:35.629177 (+ 600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:35.629178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:35.629179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:35.629183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:35.629196 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:35.629200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:35.629250 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:35.629348 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 19:16:35.629359 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:16:35.630497 8733 catalog_manager.cc:1347] Generated new cluster ID: 20d53c22a0674afa854afdf670f10e04
I20260620 19:16:35.630548 8733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:35.638935 8733 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:35.639075 8733 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:35.653811 8733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58a95636f50a4f4f8296c1c48b07c83e: Generated new TSK 0
I20260620 19:16:35.654009 8733 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:35.658396 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8651
2026-06-20T19:16:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1197 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:35 dist-test-slave-nq8r krb5kdc[8750](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:35 dist-test-slave-nq8r krb5kdc[8750](info): set up 2 sockets
Jun 20 19:16:35 dist-test-slave-nq8r krb5kdc[8750](info): commencing operation
krb5kdc: starting...
W20260620 19:16:36.600677 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:36.701674 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:45239
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:45239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:36.779472 8763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:36.779677 8763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:36.779707 8763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:36.780814 8763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:36.780879 8763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:36.780900 8763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:36.780925 8763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:36.780941 8763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:36.782362 8763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:45239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:45239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:36.782604 8763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:36.782806 8763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:36.787317 8769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:36.787325 8771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:36.787343 8768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:36.787493 8763 server_base.cc:1034] running on GCE node
I20260620 19:16:36.787801 8763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:36.788071 8763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:36.789225 8763 hybrid_clock.cc:648] HybridClock initialized: now 1781982996789206 us; error 39 us; skew 500 ppm
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8750](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982996, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:36.791581 8763 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:36.792017 8763 webserver.cc:460] Webserver started at http://127.19.66.254:39869/ using document root <none> and password file <none>
I20260620 19:16:36.792196 8763 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:36.792234 8763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:36.792325 8763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:36.793393 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "7c0328b071564e7293bce5697ca5fb9f"
format_stamp: "Formatted at 2026-06-20 19:16:36 on dist-test-slave-nq8r"
server_key: "d329d5f45c6ea1fd370018bdb16dfc7b"
server_key_iv: "e716890529152961a160c4bd6d46dc49"
server_key_version: "encryptionkey@0"
I20260620 19:16:36.793682 8763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "7c0328b071564e7293bce5697ca5fb9f"
format_stamp: "Formatted at 2026-06-20 19:16:36 on dist-test-slave-nq8r"
server_key: "d329d5f45c6ea1fd370018bdb16dfc7b"
server_key_iv: "e716890529152961a160c4bd6d46dc49"
server_key_version: "encryptionkey@0"
I20260620 19:16:36.794862 8763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:36.795503 8778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:36.795647 8763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:36.795723 8763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "7c0328b071564e7293bce5697ca5fb9f"
format_stamp: "Formatted at 2026-06-20 19:16:36 on dist-test-slave-nq8r"
server_key: "d329d5f45c6ea1fd370018bdb16dfc7b"
server_key_iv: "e716890529152961a160c4bd6d46dc49"
server_key_version: "encryptionkey@0"
I20260620 19:16:36.795805 8763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:36.808362 8763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:36.810030 8763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:36.810149 8763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:36.813820 8763 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:45239
I20260620 19:16:36.813864 8830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:45239 every 8 connection(s)
I20260620 19:16:36.814168 8763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:36.814961 8831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:36.815446 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8763
I20260620 19:16:36.815533 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:36.815713 19723 external_mini_cluster.cc:1428] Setting key f903ffde76448bd71d2a32979b47d651
I20260620 19:16:36.817828 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f: Bootstrap starting.
I20260620 19:16:36.818507 8831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:36.818818 8831 log.cc:824] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:36.819448 8831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f: No bootstrap required, opened a new log
I20260620 19:16:36.821239 8831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } }
I20260620 19:16:36.821381 8831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:36.821403 8831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c0328b071564e7293bce5697ca5fb9f, State: Initialized, Role: FOLLOWER
I20260620 19:16:36.821512 8831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } }
I20260620 19:16:36.821571 8831 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:36.821595 8831 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:36.821640 8831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:36.822142 8831 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } }
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8750](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:36.822237 8831 leader_election.cc:302] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c0328b071564e7293bce5697ca5fb9f; no voters:
I20260620 19:16:36.822398 8831 leader_election.cc:288] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:36.822433 8836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:36.822603 8836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [term 1 LEADER]: Becoming Leader. State: Replica: 7c0328b071564e7293bce5697ca5fb9f, State: Running, Role: LEADER
I20260620 19:16:36.822650 8831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:36.822765 8836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } }
I20260620 19:16:36.823073 8836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c0328b071564e7293bce5697ca5fb9f. Latest consensus state: current_term: 1 leader_uuid: "7c0328b071564e7293bce5697ca5fb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } } }
I20260620 19:16:36.823137 8836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:36.823354 8841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:36.823859 8841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:36.824175 8838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c0328b071564e7293bce5697ca5fb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c0328b071564e7293bce5697ca5fb9f" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 45239 } } }
I20260620 19:16:36.824961 8838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:36.825991 8841 catalog_manager.cc:1347] Generated new cluster ID: 3d1a907cd1b1420bbdc299a1de5883f0
I20260620 19:16:36.826071 8841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:36.826285 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:36.816209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45305 (local address 127.19.66.254:45239)
0620 19:16:36.816394 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:36.816404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:36.816441 (+ 37us) server_negotiation.cc:407] Connection header received
0620 19:16:36.816978 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:36.816987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:36.817050 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:36.817201 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:36.817758 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:36.819077 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:36.819868 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:36.820290 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:36.820381 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:36.822747 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:36.822752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:36.822752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:36.822764 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:36.824628 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:36.825099 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:36.825102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:36.825103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:36.825182 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:36.825391 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:36.825392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:36.825392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:36.825691 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:36.825833 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:36.825965 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 19:16:36.826059 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260620 19:16:36.827122 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:36.838497 8834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:36.833104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51721 (local address 127.19.66.254:45239)
0620 19:16:36.833138 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:36.833140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:36.833285 (+ 145us) server_negotiation.cc:407] Connection header received
0620 19:16:36.833358 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:36.833359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:36.833368 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:36.833418 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:36.833880 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:36.834353 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:36.834990 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:36.835190 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:36.835241 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:36.836266 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:36.836268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:36.836268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:36.836274 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:36.837501 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:36.837825 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:36.837826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:36.837827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:36.837874 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:36.838053 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:36.838054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:36.838054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:36.838152 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:36.838250 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:36.838357 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 19:16:36.838413 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:36.859503 8841 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:36.860227 8841 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:36.863202 8841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c0328b071564e7293bce5697ca5fb9f: Generated new TSK 0
I20260620 19:16:36.863379 8841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:36.867933 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8763
W20260620 19:16:36.871775 8748 connection.cc:535] client connection to 127.19.66.254:45239 recv error: Network error: failed to read from TLS socket (remote: 127.19.66.254:45239): Connection reset by peer (error 104)
2026-06-20T19:16:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1209 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8862](info): set up 2 sockets
Jun 20 19:16:36 dist-test-slave-nq8r krb5kdc[8862](info): commencing operation
krb5kdc: starting...
W20260620 19:16:37.812592 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:37 dist-test-slave-nq8r krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:37.917212 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44321
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:43621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:44321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:37.994241 8875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:37.994446 8875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:37.994477 8875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:37.995636 8875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:37.995680 8875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:37.995698 8875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:37.995713 8875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:37.995728 8875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:37.997180 8875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:43621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:44321
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:37.997427 8875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:37.997619 8875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:38.002048 8880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:38.002063 8883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:38.002067 8881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:38.002259 8875 server_base.cc:1034] running on GCE node
I20260620 19:16:38.002498 8875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:38.002774 8875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:38.003916 8875 hybrid_clock.cc:648] HybridClock initialized: now 1781982998003898 us; error 34 us; skew 500 ppm
Jun 20 19:16:38 dist-test-slave-nq8r krb5kdc[8862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982998, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:38.006242 8875 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:38.006649 8875 webserver.cc:460] Webserver started at http://127.19.66.254:39361/ using document root <none> and password file <none>
I20260620 19:16:38.006830 8875 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:38.006868 8875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:38.006956 8875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:38.007925 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "bf14db8196e247a49cd623dbeb8a1392"
format_stamp: "Formatted at 2026-06-20 19:16:38 on dist-test-slave-nq8r"
server_key: "d50ddb67a490a0dcea342f8d91c31168"
server_key_iv: "9c70b80f1bd051fcc86696b441a04323"
server_key_version: "encryptionkey@0"
I20260620 19:16:38.008198 8875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "bf14db8196e247a49cd623dbeb8a1392"
format_stamp: "Formatted at 2026-06-20 19:16:38 on dist-test-slave-nq8r"
server_key: "d50ddb67a490a0dcea342f8d91c31168"
server_key_iv: "9c70b80f1bd051fcc86696b441a04323"
server_key_version: "encryptionkey@0"
I20260620 19:16:38.009462 8875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 19:16:38.010061 8890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:38.010205 8875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:38.010286 8875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "bf14db8196e247a49cd623dbeb8a1392"
format_stamp: "Formatted at 2026-06-20 19:16:38 on dist-test-slave-nq8r"
server_key: "d50ddb67a490a0dcea342f8d91c31168"
server_key_iv: "9c70b80f1bd051fcc86696b441a04323"
server_key_version: "encryptionkey@0"
I20260620 19:16:38.010349 8875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:38.029436 8875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:38.031183 8875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:38.031316 8875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:38.035086 8875 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:44321
I20260620 19:16:38.035120 8942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:44321 every 8 connection(s)
I20260620 19:16:38.035432 8875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:38.036271 8943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:38.038571 8943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392: Bootstrap starting.
I20260620 19:16:38.039100 8943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:38.039331 8943 log.cc:824] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:38.039952 8943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392: No bootstrap required, opened a new log
I20260620 19:16:38.041167 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8875
I20260620 19:16:38.041273 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:38.041450 19723 external_mini_cluster.cc:1428] Setting key ff27f14d8eba8af6c01e05a7bbe93b42
I20260620 19:16:38.041801 8943 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } }
I20260620 19:16:38.041978 8943 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:38.042049 8943 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf14db8196e247a49cd623dbeb8a1392, State: Initialized, Role: FOLLOWER
I20260620 19:16:38.042223 8943 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } }
I20260620 19:16:38.042297 8943 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:38.042327 8943 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:38.042375 8943 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:38.043203 8943 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } }
I20260620 19:16:38.043303 8943 leader_election.cc:302] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf14db8196e247a49cd623dbeb8a1392; no voters:
I20260620 19:16:38.043449 8943 leader_election.cc:288] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:38.043491 8948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:38.043673 8948 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [term 1 LEADER]: Becoming Leader. State: Replica: bf14db8196e247a49cd623dbeb8a1392, State: Running, Role: LEADER
I20260620 19:16:38.043684 8943 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:38.043828 8948 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } }
I20260620 19:16:38.044207 8949 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf14db8196e247a49cd623dbeb8a1392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } } }
I20260620 19:16:38.044289 8949 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:38.044461 8950 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf14db8196e247a49cd623dbeb8a1392. Latest consensus state: current_term: 1 leader_uuid: "bf14db8196e247a49cd623dbeb8a1392" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf14db8196e247a49cd623dbeb8a1392" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 44321 } } }
I20260620 19:16:38.044551 8950 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:38.044910 8956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:38.045455 8956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:38.046808 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:38.041961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43949 (local address 127.19.66.254:44321)
0620 19:16:38.042119 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:38.042128 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:38.042161 (+ 33us) server_negotiation.cc:407] Connection header received
0620 19:16:38.042275 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:38.042281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:38.042324 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:38.042467 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:38.042660 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:38.043639 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:38.044714 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:38.045425 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:38.045539 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:38.046190 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:38.046194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:38.046195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:38.046389 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:38.046428 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:38.046443 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:38.046524 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:38.046603 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 19:16:38.046630 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260620 19:16:38.047214 8956 catalog_manager.cc:1347] Generated new cluster ID: 7fe206d812544d2b91339ec46a4eee3d
I20260620 19:16:38.047250 8956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:38.047677 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:38.052495 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:38.049115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53325 (local address 127.19.66.254:44321)
0620 19:16:38.049146 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:38.049148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:38.049292 (+ 144us) server_negotiation.cc:407] Connection header received
0620 19:16:38.049353 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:38.049353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:38.049362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:38.049414 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:38.049534 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:38.050217 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:38.051227 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:38.051536 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:38.051600 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:38.052261 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:38.052262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:38.052263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:38.052266 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:38.052280 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:38.052284 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:38.052331 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:38.052411 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 19:16:38.052423 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:16:38.059787 8956 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:38.060508 8956 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:38.069149 8956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf14db8196e247a49cd623dbeb8a1392: Generated new TSK 0
I20260620 19:16:38.069358 8956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:38.078059 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8875
W20260620 19:16:38.082154 8860 connection.cc:535] client connection to 127.19.66.254:44321 recv error: Network error: failed to read from TLS socket (remote: 127.19.66.254:44321): Connection reset by peer (error 104)
2026-06-20T19:16:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:38 dist-test-slave-nq8r krb5kdc[8974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:38 dist-test-slave-nq8r krb5kdc[8974](info): set up 2 sockets
Jun 20 19:16:38 dist-test-slave-nq8r krb5kdc[8974](info): commencing operation
krb5kdc: starting...
W20260620 19:16:39.024653 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:39 dist-test-slave-nq8r krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:39.131131 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43335
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:45117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:39.208127 8987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:39.208321 8987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:39.208343 8987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:39.209465 8987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:39.209504 8987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:39.209517 8987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:39.209527 8987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:39.209537 8987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:39.210951 8987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:45117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:43335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:39.211161 8987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:39.211388 8987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:39.215750 8993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:39.215764 8995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:39.215751 8992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:39.215920 8987 server_base.cc:1034] running on GCE node
I20260620 19:16:39.216195 8987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:39.216435 8987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:39.217561 8987 hybrid_clock.cc:648] HybridClock initialized: now 1781982999217551 us; error 28 us; skew 500 ppm
Jun 20 19:16:39 dist-test-slave-nq8r krb5kdc[8974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781982999, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:39.219830 8987 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:39.220237 8987 webserver.cc:460] Webserver started at http://127.19.66.254:40951/ using document root <none> and password file <none>
I20260620 19:16:39.220441 8987 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:39.220486 8987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:39.220606 8987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:39.221633 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "1127f16df0af4688a4ed59f0a5f12b70"
format_stamp: "Formatted at 2026-06-20 19:16:39 on dist-test-slave-nq8r"
server_key: "d6fde25830413164aeddbcd4f3aae31f"
server_key_iv: "b4e63174e5fe81fbc44e3100ccc3b97a"
server_key_version: "encryptionkey@0"
I20260620 19:16:39.221964 8987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "1127f16df0af4688a4ed59f0a5f12b70"
format_stamp: "Formatted at 2026-06-20 19:16:39 on dist-test-slave-nq8r"
server_key: "d6fde25830413164aeddbcd4f3aae31f"
server_key_iv: "b4e63174e5fe81fbc44e3100ccc3b97a"
server_key_version: "encryptionkey@0"
I20260620 19:16:39.223138 8987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:16:39.223753 9002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:39.223915 8987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:39.223985 8987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "1127f16df0af4688a4ed59f0a5f12b70"
format_stamp: "Formatted at 2026-06-20 19:16:39 on dist-test-slave-nq8r"
server_key: "d6fde25830413164aeddbcd4f3aae31f"
server_key_iv: "b4e63174e5fe81fbc44e3100ccc3b97a"
server_key_version: "encryptionkey@0"
I20260620 19:16:39.224048 8987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:39.227154 8987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:39.229194 8987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:39.229319 8987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:39.232785 8987 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:43335
I20260620 19:16:39.232822 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:43335 every 8 connection(s)
I20260620 19:16:39.233143 8987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:39.233999 9055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:39.234798 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 8987
I20260620 19:16:39.234892 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:39.235047 19723 external_mini_cluster.cc:1428] Setting key fcd7c8721a6b1b4e84f796fed980c935
I20260620 19:16:39.236964 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70: Bootstrap starting.
I20260620 19:16:39.237257 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:39.235483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48171 (local address 127.19.66.254:43335)
0620 19:16:39.235659 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:39.235668 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:39.235703 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:16:39.236151 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:39.236160 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:39.236195 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:39.236346 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:39.236869 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:39.236872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:39.236873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:39.236877 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:39.236917 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:39.236932 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:39.237004 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:39.237047 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 19:16:39.237084 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260620 19:16:39.237622 9055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:39.237896 9055 log.cc:824] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:39.238152 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:39.238535 9055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70: No bootstrap required, opened a new log
I20260620 19:16:39.240322 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:39.239472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45051 (local address 127.19.66.254:43335)
0620 19:16:39.239505 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:39.239507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:39.239556 (+ 49us) server_negotiation.cc:407] Connection header received
0620 19:16:39.239635 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:39.239636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:39.239639 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:39.239681 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:39.240154 (+ 473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:39.240155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:39.240155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:39.240158 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:39.240169 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:39.240172 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:39.240209 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:39.240263 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 19:16:39.240272 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:39.240662 9055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } }
I20260620 19:16:39.240793 9055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:39.240835 9055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1127f16df0af4688a4ed59f0a5f12b70, State: Initialized, Role: FOLLOWER
I20260620 19:16:39.241020 9055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } }
I20260620 19:16:39.241107 9055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:39.241145 9055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:39.241191 9055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:39.241695 9055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } }
I20260620 19:16:39.241788 9055 leader_election.cc:302] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1127f16df0af4688a4ed59f0a5f12b70; no voters:
I20260620 19:16:39.241952 9055 leader_election.cc:288] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:39.242004 9065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:39.242144 9065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [term 1 LEADER]: Becoming Leader. State: Replica: 1127f16df0af4688a4ed59f0a5f12b70, State: Running, Role: LEADER
I20260620 19:16:39.242189 9055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:39.242224 9065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } }
I20260620 19:16:39.242591 9067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1127f16df0af4688a4ed59f0a5f12b70. Latest consensus state: current_term: 1 leader_uuid: "1127f16df0af4688a4ed59f0a5f12b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } } }
I20260620 19:16:39.242601 9066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1127f16df0af4688a4ed59f0a5f12b70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1127f16df0af4688a4ed59f0a5f12b70" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 43335 } } }
I20260620 19:16:39.242764 9066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:39.242950 9073 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:39.243464 9073 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:39.242947 9067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:39.244864 9073 catalog_manager.cc:1347] Generated new cluster ID: 9d0df3fca8d44db1a4edd91a01e4ff21
I20260620 19:16:39.244913 9073 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:39.258525 9073 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:39.258666 9073 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:39.260593 9073 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1127f16df0af4688a4ed59f0a5f12b70: Generated new TSK 0
I20260620 19:16:39.260737 9073 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:39.268095 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 8987
2026-06-20T19:16:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1189 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:39 dist-test-slave-nq8r krb5kdc[9086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:39 dist-test-slave-nq8r krb5kdc[9086](info): set up 2 sockets
Jun 20 19:16:39 dist-test-slave-nq8r krb5kdc[9086](info): commencing operation
krb5kdc: starting...
W20260620 19:16:40.208508 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:40.312228 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46343
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:46343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:40.390249 9099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:40.390462 9099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:40.390493 9099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:40.391602 9099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:40.391641 9099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:40.391659 9099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:40.391674 9099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:40.391690 9099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:40.393136 9099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:46343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:46343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:40.393375 9099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:40.393568 9099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:40.397986 9104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:40.397986 9107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:40.398020 9105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:40.398023 9099 server_base.cc:1034] running on GCE node
I20260620 19:16:40.398399 9099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:40.398684 9099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:40.399832 9099 hybrid_clock.cc:648] HybridClock initialized: now 1781983000399808 us; error 31 us; skew 500 ppm
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983000, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:40.402083 9099 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:40.402489 9099 webserver.cc:460] Webserver started at http://127.19.66.254:42293/ using document root <none> and password file <none>
I20260620 19:16:40.402679 9099 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:40.402725 9099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:40.402843 9099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:40.403882 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "f51a4db091684572b27fcdf7404123d3"
format_stamp: "Formatted at 2026-06-20 19:16:40 on dist-test-slave-nq8r"
server_key: "c00bacd635efe7681f8a41384e884014"
server_key_iv: "dbd72fadcf2083c64c994bb36bef1e34"
server_key_version: "encryptionkey@0"
I20260620 19:16:40.404212 9099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "f51a4db091684572b27fcdf7404123d3"
format_stamp: "Formatted at 2026-06-20 19:16:40 on dist-test-slave-nq8r"
server_key: "c00bacd635efe7681f8a41384e884014"
server_key_iv: "dbd72fadcf2083c64c994bb36bef1e34"
server_key_version: "encryptionkey@0"
I20260620 19:16:40.405431 9099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:40.406083 9114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:40.406225 9099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:40.406299 9099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "f51a4db091684572b27fcdf7404123d3"
format_stamp: "Formatted at 2026-06-20 19:16:40 on dist-test-slave-nq8r"
server_key: "c00bacd635efe7681f8a41384e884014"
server_key_iv: "dbd72fadcf2083c64c994bb36bef1e34"
server_key_version: "encryptionkey@0"
I20260620 19:16:40.406360 9099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:40.418712 9099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:40.420440 9099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:40.420552 9099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:40.424211 9099 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:46343
I20260620 19:16:40.424243 9166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:46343 every 8 connection(s)
I20260620 19:16:40.424623 9099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:40.425480 9167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:40.426340 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9099
I20260620 19:16:40.426435 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:40.426594 19723 external_mini_cluster.cc:1428] Setting key ea2186fc1fc5cd4235a06b1264a26a3e
I20260620 19:16:40.427948 9167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3: Bootstrap starting.
I20260620 19:16:40.428648 9167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:40.428972 9167 log.cc:824] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:40.429606 9167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3: No bootstrap required, opened a new log
I20260620 19:16:40.431893 9167 raft_consensus.cc:348] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } }
I20260620 19:16:40.432072 9167 raft_consensus.cc:374] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:40.432116 9167 raft_consensus.cc:729] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f51a4db091684572b27fcdf7404123d3, State: Initialized, Role: FOLLOWER
I20260620 19:16:40.432248 9167 consensus_queue.cc:260] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } }
I20260620 19:16:40.432324 9167 raft_consensus.cc:388] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:40.432404 9167 raft_consensus.cc:482] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:40.432461 9167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:40.433166 9167 raft_consensus.cc:504] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } }
I20260620 19:16:40.433282 9167 leader_election.cc:302] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f51a4db091684572b27fcdf7404123d3; no voters:
I20260620 19:16:40.433439 9167 leader_election.cc:288] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:40.433467 9172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:40.433641 9172 raft_consensus.cc:686] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [term 1 LEADER]: Becoming Leader. State: Replica: f51a4db091684572b27fcdf7404123d3, State: Running, Role: LEADER
I20260620 19:16:40.433682 9167 sys_catalog.cc:564] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:40.433779 9172 consensus_queue.cc:237] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } }
I20260620 19:16:40.434103 9172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f51a4db091684572b27fcdf7404123d3. Latest consensus state: current_term: 1 leader_uuid: "f51a4db091684572b27fcdf7404123d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } } }
I20260620 19:16:40.434110 9174 sys_catalog.cc:455] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f51a4db091684572b27fcdf7404123d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f51a4db091684572b27fcdf7404123d3" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 46343 } } }
I20260620 19:16:40.434244 9174 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [sys.catalog]: This master's current role is: LEADER
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:40.434441 9172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:40.434885 9178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:40.435364 9178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:40.436733 9178 catalog_manager.cc:1347] Generated new cluster ID: 1c1b2cdbd0084f2f9be97c87d0975a6a
I20260620 19:16:40.436779 9178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:40.437996 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:40.427104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57733 (local address 127.19.66.254:46343)
0620 19:16:40.427256 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:40.427264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:40.427290 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:16:40.427743 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:40.427750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:40.427796 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:40.427920 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:40.428654 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:40.429660 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:40.430666 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:40.431068 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:40.431141 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:40.434796 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:40.434800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:40.434800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:40.434811 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:40.436560 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:40.436947 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:40.436951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:40.436951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:40.437048 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:40.437216 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:40.437217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:40.437218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:40.437444 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:40.437564 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:40.437682 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 19:16:40.437823 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260620 19:16:40.438810 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:40.446890 9178 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:40.447535 9178 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:40.450336 9170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:40.441118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60805 (local address 127.19.66.254:46343)
0620 19:16:40.441151 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:40.441153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:40.445002 (+ 3849us) server_negotiation.cc:407] Connection header received
0620 19:16:40.445059 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:40.445060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:40.445070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:40.445116 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:40.445580 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:40.446058 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:40.446806 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:40.447036 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:40.447087 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:40.448179 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:40.448180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:40.448180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:40.448187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:40.449416 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:40.449697 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:40.449697 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:40.449698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:40.449749 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:40.449908 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:40.449909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:40.449909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:40.450007 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:40.450084 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:40.450179 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 19:16:40.450256 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:16:40.461580 9178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f51a4db091684572b27fcdf7404123d3: Generated new TSK 0
I20260620 19:16:40.461768 9178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:40.464557 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9099
2026-06-20T19:16:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9198](info): set up 2 sockets
Jun 20 19:16:40 dist-test-slave-nq8r krb5kdc[9198](info): commencing operation
krb5kdc: starting...
W20260620 19:16:41.408538 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:41 dist-test-slave-nq8r krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:41.511497 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:41229
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:43623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:41.590937 9211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:41.591136 9211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:41.591158 9211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:41.592303 9211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:41.592344 9211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:41.592357 9211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:41.592370 9211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:41.592383 9211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:41.593892 9211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:43623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:41.594116 9211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:41.594319 9211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:41.598727 9216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:41.598807 9217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:41.598821 9211 server_base.cc:1034] running on GCE node
W20260620 19:16:41.598812 9219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:41.599174 9211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:41.599433 9211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:41.600579 9211 hybrid_clock.cc:648] HybridClock initialized: now 1781983001600565 us; error 28 us; skew 500 ppm
Jun 20 19:16:41 dist-test-slave-nq8r krb5kdc[9198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983001, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:41.606907 9211 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:41.607282 9211 webserver.cc:460] Webserver started at http://127.19.66.254:35257/ using document root <none> and password file <none>
I20260620 19:16:41.607474 9211 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:41.607517 9211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:41.607632 9211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:41.608650 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "11552bc48b5245e19838930192ff5285"
format_stamp: "Formatted at 2026-06-20 19:16:41 on dist-test-slave-nq8r"
server_key: "b72515bc63ba25fae219165af1fafe5e"
server_key_iv: "d7162ba20d292517d1748749537640ec"
server_key_version: "encryptionkey@0"
I20260620 19:16:41.608994 9211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "11552bc48b5245e19838930192ff5285"
format_stamp: "Formatted at 2026-06-20 19:16:41 on dist-test-slave-nq8r"
server_key: "b72515bc63ba25fae219165af1fafe5e"
server_key_iv: "d7162ba20d292517d1748749537640ec"
server_key_version: "encryptionkey@0"
I20260620 19:16:41.610175 9211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 19:16:41.610802 9226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:41.610947 9211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:41.611022 9211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "11552bc48b5245e19838930192ff5285"
format_stamp: "Formatted at 2026-06-20 19:16:41 on dist-test-slave-nq8r"
server_key: "b72515bc63ba25fae219165af1fafe5e"
server_key_iv: "d7162ba20d292517d1748749537640ec"
server_key_version: "encryptionkey@0"
I20260620 19:16:41.611083 9211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:41.625346 9211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:41.627365 9211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:41.627491 9211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:41.631162 9211 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:41229
I20260620 19:16:41.631206 9278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:41229 every 8 connection(s)
I20260620 19:16:41.631502 9211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:41.632277 9279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:41.634497 9279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: Bootstrap starting.
I20260620 19:16:41.635012 9279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:41.635227 9279 log.cc:824] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:41.635852 9279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: No bootstrap required, opened a new log
I20260620 19:16:41.635908 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9211
I20260620 19:16:41.636003 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:41.636186 19723 external_mini_cluster.cc:1428] Setting key 9d0f3f9649900fd0c8333c70dbd0d474
I20260620 19:16:41.637847 9279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } }
I20260620 19:16:41.638019 9279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:41.638092 9279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11552bc48b5245e19838930192ff5285, State: Initialized, Role: FOLLOWER
I20260620 19:16:41.638226 9279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } }
I20260620 19:16:41.638310 9279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:41.638347 9279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:41.638393 9279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:41.639071 9279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } }
I20260620 19:16:41.639176 9279 leader_election.cc:302] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11552bc48b5245e19838930192ff5285; no voters:
I20260620 19:16:41.639317 9279 leader_election.cc:288] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:41.639353 9284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:41.639477 9284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [term 1 LEADER]: Becoming Leader. State: Replica: 11552bc48b5245e19838930192ff5285, State: Running, Role: LEADER
I20260620 19:16:41.639539 9279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:41.639595 9284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } }
I20260620 19:16:41.639910 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11552bc48b5245e19838930192ff5285. Latest consensus state: current_term: 1 leader_uuid: "11552bc48b5245e19838930192ff5285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } } }
I20260620 19:16:41.640098 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:41.640041 9286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11552bc48b5245e19838930192ff5285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11552bc48b5245e19838930192ff5285" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 41229 } } }
I20260620 19:16:41.640439 9286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285 [sys.catalog]: This master's current role is: LEADER
W20260620 19:16:41.641180 9300 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:41.641232 9300 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:41.641273 9293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:41.641430 9293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:41.642802 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:41.636674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54073 (local address 127.19.66.254:41229)
0620 19:16:41.636870 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:41.636879 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:41.636934 (+ 55us) server_negotiation.cc:407] Connection header received
0620 19:16:41.637807 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:41.637816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:41.637881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:41.638023 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:41.638186 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:41.639833 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:41.640954 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:41.641437 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:41.641794 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:41.642159 (+ 365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:41.642163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:41.642164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:41.642349 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:41.642388 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:41.642403 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:41.642462 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:41.642520 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 19:16:41.642579 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260620 19:16:41.642845 9293 catalog_manager.cc:1347] Generated new cluster ID: 6738ce33310f4809a6c20b001691450f
I20260620 19:16:41.642886 9293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:41.645629 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:41.649398 9283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:41.647070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52703 (local address 127.19.66.254:41229)
0620 19:16:41.647104 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:41.647105 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:41.647112 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:41.647128 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:41.647129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:41.647138 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:41.647185 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:41.647301 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:41.647747 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:41.648394 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:41.648586 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:41.648636 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:41.649204 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:41.649205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:41.649205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:41.649210 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:41.649224 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:41.649228 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:41.649270 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:41.649312 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 19:16:41.649333 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:16:41.658809 9293 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:41.659497 9293 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:41.668926 9293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11552bc48b5245e19838930192ff5285: Generated new TSK 0
I20260620 19:16:41.669100 9293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:41.679345 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9211
2026-06-20T19:16:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:41 dist-test-slave-nq8r krb5kdc[9310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:41 dist-test-slave-nq8r krb5kdc[9310](info): set up 2 sockets
Jun 20 19:16:41 dist-test-slave-nq8r krb5kdc[9310](info): commencing operation
krb5kdc: starting...
W20260620 19:16:42.616621 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:42 dist-test-slave-nq8r krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:42.724215 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:42651
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:38943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:42651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:42.802168 9323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:42.802376 9323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:42.802405 9323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:42.803512 9323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:42.803552 9323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:42.803570 9323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:42.803586 9323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:42.803601 9323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:42.805074 9323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:38943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:42651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:42651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:42.805305 9323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:42.805498 9323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:42.809928 9331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:42.809928 9328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:42.810103 9323 server_base.cc:1034] running on GCE node
W20260620 19:16:42.809911 9329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:42.810352 9323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:42.810607 9323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:42.811743 9323 hybrid_clock.cc:648] HybridClock initialized: now 1781983002811728 us; error 25 us; skew 500 ppm
Jun 20 19:16:42 dist-test-slave-nq8r krb5kdc[9310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983002, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:42.813905 9323 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:42.814298 9323 webserver.cc:460] Webserver started at http://127.19.66.254:34413/ using document root <none> and password file <none>
I20260620 19:16:42.814495 9323 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:42.814540 9323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:42.814658 9323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:42.815630 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "a7cdcdfa88034abfbd0e7f765acd01f1"
format_stamp: "Formatted at 2026-06-20 19:16:42 on dist-test-slave-nq8r"
server_key: "f9f99f869d0965795b121ddfffc83d72"
server_key_iv: "684a383eaf9555be231fc3c566c2ba4a"
server_key_version: "encryptionkey@0"
I20260620 19:16:42.815943 9323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "a7cdcdfa88034abfbd0e7f765acd01f1"
format_stamp: "Formatted at 2026-06-20 19:16:42 on dist-test-slave-nq8r"
server_key: "f9f99f869d0965795b121ddfffc83d72"
server_key_iv: "684a383eaf9555be231fc3c566c2ba4a"
server_key_version: "encryptionkey@0"
I20260620 19:16:42.817118 9323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:42.817826 9338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:42.817982 9323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:42.818082 9323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "a7cdcdfa88034abfbd0e7f765acd01f1"
format_stamp: "Formatted at 2026-06-20 19:16:42 on dist-test-slave-nq8r"
server_key: "f9f99f869d0965795b121ddfffc83d72"
server_key_iv: "684a383eaf9555be231fc3c566c2ba4a"
server_key_version: "encryptionkey@0"
I20260620 19:16:42.818136 9323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:42.821209 9323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:42.822829 9323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:42.822952 9323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:42.826488 9323 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:42651
I20260620 19:16:42.826516 9390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:42651 every 8 connection(s)
I20260620 19:16:42.826803 9323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:42.827677 9391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:42.828126 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9323
I20260620 19:16:42.828227 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:42.828377 19723 external_mini_cluster.cc:1428] Setting key d3d3b5acb7234f53713837f5d5e21758
I20260620 19:16:42.830406 9391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1: Bootstrap starting.
I20260620 19:16:42.830667 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:42.828834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50221 (local address 127.19.66.254:42651)
0620 19:16:42.829026 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:42.829036 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:42.829070 (+ 34us) server_negotiation.cc:407] Connection header received
0620 19:16:42.829564 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:42.829572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:42.829605 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:42.829755 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:42.830280 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:42.830283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:42.830284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:42.830288 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:42.830327 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:42.830341 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:42.830405 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:42.830462 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 19:16:42.830500 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 19:16:42.831068 9391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:42.831354 9391 log.cc:824] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:42.831543 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:42.831962 9391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1: No bootstrap required, opened a new log
I20260620 19:16:42.833719 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:42.832795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38015 (local address 127.19.66.254:42651)
0620 19:16:42.832866 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:42.832867 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:42.832948 (+ 81us) server_negotiation.cc:407] Connection header received
0620 19:16:42.832983 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:42.832984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:42.832998 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:42.833062 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:42.833521 (+ 459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:42.833522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:42.833523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 19:16:42.833529 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:42.833542 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 19:16:42.833546 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:42.833611 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:42.833650 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 19:16:42.833661 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:42.834072 9391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } }
I20260620 19:16:42.834201 9391 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 19:16:42.834232 9391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cdcdfa88034abfbd0e7f765acd01f1, State: Initialized, Role: FOLLOWER
I20260620 19:16:42.834369 9391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } }
I20260620 19:16:42.834436 9391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 19:16:42.834473 9391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 19:16:42.834517 9391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:42.834980 9391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } }
I20260620 19:16:42.835065 9391 leader_election.cc:302] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7cdcdfa88034abfbd0e7f765acd01f1; no voters:
I20260620 19:16:42.835204 9391 leader_election.cc:288] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 19:16:42.835258 9401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:42.835391 9401 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [term 1 LEADER]: Becoming Leader. State: Replica: a7cdcdfa88034abfbd0e7f765acd01f1, State: Running, Role: LEADER
I20260620 19:16:42.835415 9391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:42.835513 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } }
I20260620 19:16:42.835835 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } } }
I20260620 19:16:42.835846 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7cdcdfa88034abfbd0e7f765acd01f1. Latest consensus state: current_term: 1 leader_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7cdcdfa88034abfbd0e7f765acd01f1" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 42651 } } }
I20260620 19:16:42.836005 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:42.836180 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:42.836467 9409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:42.836939 9409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:42.838250 9409 catalog_manager.cc:1347] Generated new cluster ID: b6d39c7950ed4e9599811d4d29501fca
I20260620 19:16:42.838299 9409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:42.847831 9409 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:42.847970 9409 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:42.858991 9409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7cdcdfa88034abfbd0e7f765acd01f1: Generated new TSK 0
I20260620 19:16:42.859153 9409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:42.862958 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9323
2026-06-20T19:16:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1183 ms)
[----------] 18 tests from AuthTokenIssuingTest (21600 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:42 dist-test-slave-nq8r krb5kdc[9422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:42 dist-test-slave-nq8r krb5kdc[9422](info): set up 2 sockets
Jun 20 19:16:42 dist-test-slave-nq8r krb5kdc[9422](info): commencing operation
krb5kdc: starting...
W20260620 19:16:43.812626 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:43 dist-test-slave-nq8r krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:43.916594 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38317
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:43.993062 9435 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:43.993253 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:43.993268 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:43.993286 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:43.994398 9435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:43.994436 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:43.994448 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:43.994463 9435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:43.994486 9435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:43.995924 9435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:38317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:43.996148 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:43.996342 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:44.000639 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:44.000717 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:44.000773 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:44.001025 9435 server_base.cc:1034] running on GCE node
I20260620 19:16:44.001221 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:44.001456 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:44.002614 9435 hybrid_clock.cc:648] HybridClock initialized: now 1781983004002594 us; error 30 us; skew 500 ppm
Jun 20 19:16:43 dist-test-slave-nq8r krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:44.004998 9435 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:44.005395 9435 webserver.cc:460] Webserver started at http://127.19.66.254:43445/ using document root <none> and password file <none>
I20260620 19:16:44.005614 9435 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:44.005661 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:44.005797 9435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:44.006808 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "19b35764c513ee18cbda233dde628698"
server_key_iv: "2b6b08d974b46c682750f87f9a1b0a07"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.007138 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "19b35764c513ee18cbda233dde628698"
server_key_iv: "2b6b08d974b46c682750f87f9a1b0a07"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.008293 9435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:44.008958 9450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:44.009122 9435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:44.009210 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "19b35764c513ee18cbda233dde628698"
server_key_iv: "2b6b08d974b46c682750f87f9a1b0a07"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.009275 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:44.022601 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:44.024192 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:44.024317 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:44.027952 9435 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:38317
I20260620 19:16:44.027990 9502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:38317 every 8 connection(s)
I20260620 19:16:44.028294 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:44.029219 9503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:44.030371 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9435
I20260620 19:16:44.030469 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:44.030618 19723 external_mini_cluster.cc:1428] Setting key 33997d4eef39c432e1f00917f448acb2
I20260620 19:16:44.030701 9503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.253@KRBTEST.COM" created.
I20260620 19:16:44.039507 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.031388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38317 (local address 127.0.0.1:55790)
0620 19:16:44.031561 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.031597 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.032067 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.032308 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.032311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.032568 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.032711 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.032715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.033685 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.033687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.034346 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.034349 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.034795 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.034796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.034864 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.035290 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.035298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.036532 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.038071 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.038077 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.038079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.038309 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.038508 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.038509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.038510 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.038622 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.038962 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.038967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.039029 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.039108 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.039330 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":103,"thread_start_us":96,"threads_started":1}
I20260620 19:16:44.039507 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.031399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55790 (local address 127.19.66.254:38317)
0620 19:16:44.031551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.031561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.031679 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:16:44.032203 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.032205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.032221 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.032293 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.032793 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.033605 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.034420 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.034712 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.034804 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.036606 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.036612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.036612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.036619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.037998 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.038381 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.038384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.038384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.038455 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.038682 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.038683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.038683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.038904 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.038972 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.039147 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.039277 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260620 19:16:44.040561 9503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:44.041807 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.041692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57630)
0620 19:16:44.041739 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.041783 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 19:16:44.041937 9454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
I20260620 19:16:44.042320 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.042241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57634)
0620 19:16:44.042281 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.042304 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 19:16:44.042416 9503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:40791: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
I20260620 19:16:44.051342 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:40791
--webserver_interface=127.19.66.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:16:44.091671 9503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } attempt: 1
I20260620 19:16:44.092180 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.092058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57648)
0620 19:16:44.092116 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.092148 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:16:44.092533 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.092463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57660)
0620 19:16:44.092496 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.092517 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:16:44.092613 9503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:40791: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
W20260620 19:16:44.128559 9510 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:44.128759 9510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:44.128777 9510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:44.128795 9510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:44.129873 9510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:44.129906 9510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:44.129918 9510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:44.129930 9510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:44.129942 9510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:44.131299 9510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:40791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.66.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:44.131482 9510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:44.131642 9510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:44.136039 9515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:44.136098 9510 server_base.cc:1034] running on GCE node
W20260620 19:16:44.136047 9516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:44.136047 9518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:44.136408 9510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:44.136653 9510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:44.137784 9510 hybrid_clock.cc:648] HybridClock initialized: now 1781983004137752 us; error 46 us; skew 500 ppm
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:44.139997 9510 init.cc:377] Logged in from keytab as kudu/127.19.66.253@KRBTEST.COM (short username kudu)
I20260620 19:16:44.140399 9510 webserver.cc:460] Webserver started at http://127.19.66.253:34625/ using document root <none> and password file <none>
I20260620 19:16:44.140583 9510 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:44.140625 9510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:44.140748 9510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:44.141760 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data/instance:
uuid: "6b767e7f61664112abf28c6b7fa92a80"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "2760040f704af5a83c0ccb0d43e623f2"
server_key_iv: "7a3c72c101c69ccd9c1dda1c3a5c788c"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.142078 9510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal/instance:
uuid: "6b767e7f61664112abf28c6b7fa92a80"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "2760040f704af5a83c0ccb0d43e623f2"
server_key_iv: "7a3c72c101c69ccd9c1dda1c3a5c788c"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.143205 9510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:16:44.143798 9525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:44.143940 9510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:44.144014 9510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "6b767e7f61664112abf28c6b7fa92a80"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "2760040f704af5a83c0ccb0d43e623f2"
server_key_iv: "7a3c72c101c69ccd9c1dda1c3a5c788c"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.144078 9510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:44.160774 9503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } attempt: 2
I20260620 19:16:44.161224 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.161107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57668)
0620 19:16:44.161165 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.161196 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:16:44.161500 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.161423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57682)
0620 19:16:44.161456 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.161478 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:16:44.161571 9503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:40791: Network error: Client connection negotiation failed: client connection to 127.19.66.253:40791: connect: Connection refused (error 111)
I20260620 19:16:44.171716 9510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:44.173520 9510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:44.173647 9510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:44.177284 9510 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.253:40791
I20260620 19:16:44.177314 9577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.253:40791 every 8 connection(s)
I20260620 19:16:44.177627 9510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
I20260620 19:16:44.178495 9578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:44.179970 9578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:44.185436 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9510
I20260620 19:16:44.185532 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-1/wal/instance
I20260620 19:16:44.185700 19723 external_mini_cluster.cc:1428] Setting key 0d4a2e255a60df821626e12769cc09d8
I20260620 19:16:44.187918 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.180592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.19.66.254:38317)
0620 19:16:44.180668 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.180670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.180876 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:16:44.181361 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.181362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.181375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.181437 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.181997 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.182458 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.183342 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.183534 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.183585 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.185527 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.185529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.185530 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.185537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.186719 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.187128 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.187129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.187129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.187186 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.187373 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.187373 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.187373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.187496 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.187558 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.187754 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.187841 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 19:16:44.188004 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.180562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38317 (local address 127.0.0.1:55800)
0620 19:16:44.180728 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.180768 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.181251 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.181495 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.181498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.181728 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.181924 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.181928 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.182517 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.182517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.183274 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.183276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.183598 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.183599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.183695 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.184122 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.184131 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.185438 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.186800 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.186808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.186810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.187066 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.187239 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.187239 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.187239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.187325 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.187548 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.187551 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.187649 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.187720 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.187827 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260620 19:16:44.188474 9578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.252@KRBTEST.COM" created.
I20260620 19:16:44.195294 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.188678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57686)
0620 19:16:44.188719 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.188729 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.188769 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.189050 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.189051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.189160 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.189208 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.189209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.190387 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.190387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.190856 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.190857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.191420 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.191420 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.191466 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.191787 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.191792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.192824 (+ 1032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.194301 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.194302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.194302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.194452 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.194641 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.194642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.194642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.194691 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.194977 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.194978 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.195020 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.195090 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.195180 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:16:44.195317 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.188727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.19.66.253:40791)
0620 19:16:44.188901 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.188905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.188929 (+ 24us) server_negotiation.cc:407] Connection header received
0620 19:16:44.188950 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.188952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.188965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.189039 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.189272 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.190314 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.190919 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.191355 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.191427 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.192926 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.192933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.192934 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.192941 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.194199 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.194516 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.194518 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.194518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.194594 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.194742 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.194742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.194743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.194920 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.194998 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.195124 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.195226 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260620 19:16:44.195964 9578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:44.196250 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.196162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36008)
0620 19:16:44.196192 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.196233 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 19:16:44.196372 9526 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
I20260620 19:16:44.196724 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.196647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36024)
0620 19:16:44.196681 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.196703 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:16:44.196813 9578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:45391: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
I20260620 19:16:44.206526 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:45391
--webserver_interface=127.19.66.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:16:44.246061 9578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } attempt: 1
I20260620 19:16:44.246531 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.246396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36032)
0620 19:16:44.246454 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.246497 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 19:16:44.246843 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.246771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36036)
0620 19:16:44.246805 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.246827 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260620 19:16:44.246934 9578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:45391: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
I20260620 19:16:44.252696 9503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } attempt: 3
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:44.259639 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.253004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57696)
0620 19:16:44.253063 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.253074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.253132 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.253277 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.253277 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.253420 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.253479 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.253481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.254376 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.254377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.254885 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.254886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.255356 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.255356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.255405 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.255796 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.255800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.257105 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.258754 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.258755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.258755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.258908 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.259077 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.259078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.259078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.259121 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.259342 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.259342 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.259387 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.259453 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.259542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:16:44.259647 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.253049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.19.66.253:40791)
0620 19:16:44.253097 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.253100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.253114 (+ 14us) server_negotiation.cc:407] Connection header received
0620 19:16:44.253196 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.253197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.253209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.253261 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.253536 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.254305 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.254952 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.255287 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.255354 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.257193 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.257195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.257195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.257203 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.258683 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.258968 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.258969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.258970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.259027 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.259170 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.259170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.259171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.259289 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.259344 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.259500 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.259568 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:16:44.259984 9503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:44.260269 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.260193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36040)
0620 19:16:44.260224 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.260251 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 19:16:44.260586 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.260515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36050)
0620 19:16:44.260549 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.260570 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 19:16:44.260663 9503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:45391: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
W20260620 19:16:44.285483 9585 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:44.285682 9585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:44.285706 9585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:44.285732 9585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:44.286830 9585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:44.286870 9585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:44.286890 9585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:44.286906 9585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:44.286921 9585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:44.288352 9585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:38317,127.19.66.253:40791,127.19.66.252:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.66.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:44.288591 9585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:44.288784 9585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 19:16:44.291800 9503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } attempt: 1
I20260620 19:16:44.292205 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.292083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36056)
0620 19:16:44.292138 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.292171 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 19:16:44.292713 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.292622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36068)
0620 19:16:44.292661 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.292686 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:16:44.292831 9503 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:45391: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
W20260620 19:16:44.293170 9590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:44.293196 9591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:44.293177 9593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:44.293342 9585 server_base.cc:1034] running on GCE node
I20260620 19:16:44.293545 9585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:44.293810 9585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:44.294951 9585 hybrid_clock.cc:648] HybridClock initialized: now 1781983004294937 us; error 29 us; skew 500 ppm
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:44.297189 9585 init.cc:377] Logged in from keytab as kudu/127.19.66.252@KRBTEST.COM (short username kudu)
I20260620 19:16:44.297607 9585 webserver.cc:460] Webserver started at http://127.19.66.252:35369/ using document root <none> and password file <none>
I20260620 19:16:44.297777 9585 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:44.297817 9585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:44.297909 9585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:44.298938 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data/instance:
uuid: "378e9d7e1ac54c47a64b6a759a89bc9c"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "144b24d246c53b3d16556cec30cbbf96"
server_key_iv: "a6c9d87515e3f1dd6b3f99325c240d75"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.299217 9585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal/instance:
uuid: "378e9d7e1ac54c47a64b6a759a89bc9c"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "144b24d246c53b3d16556cec30cbbf96"
server_key_iv: "a6c9d87515e3f1dd6b3f99325c240d75"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.300374 9585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 19:16:44.300973 9600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:44.301115 9585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:44.301180 9585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "378e9d7e1ac54c47a64b6a759a89bc9c"
format_stamp: "Formatted at 2026-06-20 19:16:44 on dist-test-slave-nq8r"
server_key: "144b24d246c53b3d16556cec30cbbf96"
server_key_iv: "a6c9d87515e3f1dd6b3f99325c240d75"
server_key_version: "encryptionkey@0"
I20260620 19:16:44.301247 9585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:44.315078 9578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } attempt: 2
I20260620 19:16:44.315431 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.315325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36076)
0620 19:16:44.315372 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.315406 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 19:16:44.315675 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.315613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36088)
0620 19:16:44.315641 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.315661 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 19:16:44.315737 9578 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:45391: Network error: Client connection negotiation failed: client connection to 127.19.66.252:45391: connect: Connection refused (error 111)
I20260620 19:16:44.324503 9585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:44.326550 9585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:44.326674 9585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:44.330276 9585 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.252:45391
I20260620 19:16:44.330320 9652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.252:45391 every 8 connection(s)
I20260620 19:16:44.330623 9585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
I20260620 19:16:44.331543 9653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:44.332979 9653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:44.340451 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9585
I20260620 19:16:44.340554 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781982751812665-19723-0/minicluster-data/master-2/wal/instance
I20260620 19:16:44.340710 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.333616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55812 (local address 127.19.66.254:38317)
0620 19:16:44.333668 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.333670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.333845 (+ 175us) server_negotiation.cc:407] Connection header received
0620 19:16:44.334301 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.334302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.334314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.334360 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.334950 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.335407 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.336324 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.336516 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.336571 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.338440 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.338442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.338442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.338450 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.339461 (+ 1011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.339794 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.339795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.339796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.339858 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.340069 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.340070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.340070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.340174 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.340226 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.340545 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.340635 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 19:16:44.340731 19723 external_mini_cluster.cc:1428] Setting key 3e610ef86cef11173c7f46c61ae195bc
I20260620 19:16:44.340888 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.333578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:38317 (local address 127.0.0.1:55812)
0620 19:16:44.333739 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.333772 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.334213 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.334438 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.334441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.334682 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.334868 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.334871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.335483 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.335484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.336245 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.336247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.336593 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.336594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.336711 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.337131 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.337138 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.338355 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.339529 (+ 1174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.339535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.339537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.339733 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.339913 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.339913 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.339914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.340013 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.340225 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.340231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.340388 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.340500 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.340658 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260620 19:16:44.341377 9653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:44.347581 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.341636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.19.66.253:40791)
0620 19:16:44.341692 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.341694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.341701 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:44.341724 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.341725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.341733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.341773 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.342039 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.342483 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.343362 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.343667 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.343728 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.345352 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.345354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.345354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.345361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.346610 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.346924 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.346926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.346926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.346979 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.347133 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.347134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.347134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.347241 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.347294 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.347420 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.347491 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:44.347600 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.341574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:40791 (local address 127.0.0.1:57712)
0620 19:16:44.341616 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.341624 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.341657 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.341785 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.341786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.341922 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.341973 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.341975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.342546 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.342547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.343289 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.343290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.343733 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.343734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.343782 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.344104 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.344109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.345279 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.346684 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.346685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.346685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.346867 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.347030 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.347030 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.347031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.347084 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.347292 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.347293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.347332 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.347392 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.347491 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":105}
I20260620 19:16:44.347935 9653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:44.349088 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.341103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.19.66.252:45391)
0620 19:16:44.341255 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.341260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.341332 (+ 72us) server_negotiation.cc:407] Connection header received
0620 19:16:44.341402 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.341405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.341419 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.341484 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.342148 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.343005 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.343871 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.344249 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.344318 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.346478 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.346486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.346486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.346491 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.348016 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.348370 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.348373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.348374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.348428 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.348587 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.348588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.348588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.348721 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.348882 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.348926 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.348997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:44.355043 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.348142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36102)
0620 19:16:44.348179 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.348188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.348220 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.348431 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.348431 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.348544 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.348567 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.348568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.349226 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.349227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.349766 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.349768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.350358 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.350358 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.350394 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.350706 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.350709 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.351751 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.354127 (+ 2376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.354128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.354129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.354319 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.354512 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.354513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.354513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.354573 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.354823 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.354823 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.354861 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.354908 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.354981 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 19:16:44.355168 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.348202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.19.66.252:45391)
0620 19:16:44.348342 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.348343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.348349 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:16:44.348361 (+ 12us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.348362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.348368 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.348423 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.348621 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.349147 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.349987 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.350297 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.350363 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.351818 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.351819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.351819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.351827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.354052 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.354389 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.354390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.354391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.354447 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.354631 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.354631 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.354632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.354756 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.354819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.354945 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.355032 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260620 19:16:44.356315 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.349822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.19.66.254:38317)
0620 19:16:44.349868 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.349869 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.349875 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:16:44.349895 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.349896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.349904 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.349948 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.350448 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.350992 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.351660 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.351883 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.351948 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.353861 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.353862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.353863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.353871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.355371 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.355647 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.355648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.355649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.355705 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.355855 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.355856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.355857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.355963 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.356125 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.356160 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.356229 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 19:16:44.356683 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: Bootstrap starting.
I20260620 19:16:44.357409 9653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:44.357694 9653 log.cc:824] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:44.358366 9653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: No bootstrap required, opened a new log
I20260620 19:16:44.359719 9653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.359841 9653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 378e9d7e1ac54c47a64b6a759a89bc9c, State: Initialized, Role: FOLLOWER
I20260620 19:16:44.359920 9653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.360209 9653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:44.360982 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.361070 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: This master's current role is: FOLLOWER
W20260620 19:16:44.362038 9671 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:44.362092 9671 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:44.363844 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.356915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57714 (local address 127.19.66.253:40791)
0620 19:16:44.356963 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.356965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.356997 (+ 32us) server_negotiation.cc:407] Connection header received
0620 19:16:44.357060 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.357061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.357071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.357124 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.357508 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.358042 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.358684 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.358909 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.358963 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.361162 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.361163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.361163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.361170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.362777 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.363141 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.363142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.363142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.363199 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.363374 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.363374 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.363375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.363498 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.363647 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.363681 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.363751 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:16:44.362370 9585 master_runner.cc:186] Error getting master registration for 127.19.66.254:38317: OK, instance_id { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" instance_seqno: 1781983004026927 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:16:44.364934 9585 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:16:44.365272 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:44.366295 9477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55818
I20260620 19:16:44.367982 9503 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } attempt: 2
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983003, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:44.374919 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.368292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.19.66.252:45391)
0620 19:16:44.368332 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.368333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.368341 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:44.368389 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.368389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.368400 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.368443 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.368693 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.369488 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.370413 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.370625 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.370678 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.372534 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.372535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.372536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.372543 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.373840 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.374178 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.374178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.374179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.374231 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.374398 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.374399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.374399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.374561 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.374619 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.374755 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.374832 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:16:44.374920 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.368202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36118)
0620 19:16:44.368262 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.368272 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.368309 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.368458 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.368459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.368591 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.368629 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.368631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.369546 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.369547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.370347 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.370349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.370684 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.370685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.370740 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.371110 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.371114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.372447 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.373906 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.373907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.373907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.374114 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.374287 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.374288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.374288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.374347 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.374618 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.374619 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.374665 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.374730 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.374823 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 19:16:44.376430 9503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8: Bootstrap starting.
I20260620 19:16:44.377092 9503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:44.377339 9503 log.cc:824] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:44.377993 9503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8: No bootstrap required, opened a new log
I20260620 19:16:44.379271 9503 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.379403 9503 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6401bb6c544fee86b3d6e8f0beb7f8, State: Initialized, Role: FOLLOWER
I20260620 19:16:44.379484 9503 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.379787 9503 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:44.379807 9477 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:16:44.379752 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.379868 9477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:44.379869 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.380659 9477 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.380940 9477 leader_election.cc:288] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [CANDIDATE]: Term 1 election: Requested vote from peers 6b767e7f61664112abf28c6b7fa92a80 (127.19.66.253:40791), 378e9d7e1ac54c47a64b6a759a89bc9c (127.19.66.252:45391)
I20260620 19:16:44.381374 9627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c"
I20260620 19:16:44.381498 9627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:44.382344 9627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa6401bb6c544fee86b3d6e8f0beb7f8 in term 1.
I20260620 19:16:44.382642 9451 leader_election.cc:302] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 378e9d7e1ac54c47a64b6a759a89bc9c, aa6401bb6c544fee86b3d6e8f0beb7f8; no voters:
I20260620 19:16:44.383307 9674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:44.383436 9674 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [term 1 LEADER]: Becoming Leader. State: Replica: aa6401bb6c544fee86b3d6e8f0beb7f8, State: Running, Role: LEADER
I20260620 19:16:44.383524 9674 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.383924 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6401bb6c544fee86b3d6e8f0beb7f8. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.383993 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:44.384732 9435 master_runner.cc:186] Error getting master registration for 127.19.66.253:40791: OK, instance_id { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" instance_seqno: 1781983004176254 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:16:44.387507 9435 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:16:44.387809 9695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:44.389281 9695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:44.391002 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.382219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.19.66.253:40791)
0620 19:16:44.382262 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.382263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.382612 (+ 349us) server_negotiation.cc:407] Connection header received
0620 19:16:44.383045 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.383047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.383058 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.383086 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.383941 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.384614 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.385583 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.386311 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.386362 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.388086 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.388087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.388087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.388095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.389681 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.390115 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.390116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.390117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.390173 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.390383 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.390384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.390384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.390518 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.390608 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.390824 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.390907 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:16:44.393270 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.381945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.19.66.254:38317)
0620 19:16:44.382034 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.382037 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.382414 (+ 377us) server_negotiation.cc:407] Connection header received
0620 19:16:44.382508 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.382509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.382519 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.382548 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.383627 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.384349 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.386931 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.387188 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.387232 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.389958 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.389959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.389960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.389967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.391436 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.392411 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.392412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.392413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.392472 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.392670 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.392671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.392672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.392798 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.392902 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.393109 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.393183 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 19:16:44.394860 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.381982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36122 (local address 127.19.66.252:45391)
0620 19:16:44.382065 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.382067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.382445 (+ 378us) server_negotiation.cc:407] Connection header received
0620 19:16:44.382656 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.382657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.382666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.382693 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.385367 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.386036 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.387687 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.387987 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.388033 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.392121 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.392123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.392123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.392132 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.393694 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.394090 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.394091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.394092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.394147 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.394320 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.394321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.394322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.394458 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.394564 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.394699 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.394768 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 19:16:44.395808 9627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [term 1 FOLLOWER]: Refusing update from remote peer aa6401bb6c544fee86b3d6e8f0beb7f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:44.396142 9674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:44.397468 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6401bb6c544fee86b3d6e8f0beb7f8. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.397554 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.398751 9660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.398829 9660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.399014 9698 mvcc.cc:205] Tried to move back new op lower bound from 7299002385997844480 to 7299002385955594240. Current Snapshot: MvccSnapshot[applied={T|T < 7299002385997844480}]
I20260620 19:16:44.398504 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.399364 9697 mvcc.cc:205] Tried to move back new op lower bound from 7299002385997844480 to 7299002385955594240. Current Snapshot: MvccSnapshot[applied={T|T < 7299002385997844480}]
I20260620 19:16:44.399412 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:44.399861 9695 catalog_manager.cc:1347] Generated new cluster ID: 5770bdbfe294442db89b38568553a20a
I20260620 19:16:44.399907 9695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:44.406855 9578 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } attempt: 3
Jun 20 19:16:44 dist-test-slave-nq8r krb5kdc[9422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983004, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:44.413602 9695 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:44.414470 9695 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:44.415014 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.407085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:45391 (local address 127.0.0.1:36126)
0620 19:16:44.407181 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:44.407190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:44.407227 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:44.407426 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:44.407428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:44.407597 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:44.407642 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.407644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.408470 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.408472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.409293 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:44.409296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.409711 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.409711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:44.409758 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.410164 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:44.410169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:44.411653 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:44.413562 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.413563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.413564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.413773 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.414390 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:44.414391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:44.414391 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:44.414457 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:44.414719 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:44.414720 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:44.414763 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:16:44.414820 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 19:16:44.414914 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 19:16:44.415349 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:44.407162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.19.66.252:45391)
0620 19:16:44.407254 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.407256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.407263 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:44.407278 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.407279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.407288 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.407340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.407711 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.408415 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.409371 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:44.409652 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:44.409716 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:44.411740 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:44.411741 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:44.411741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:44.411748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:44.413496 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.414254 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.414256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.414256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.414322 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:44.414522 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:44.414522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:44.414523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:44.414646 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:44.414696 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:44.415187 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 19:16:44.415265 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 19:16:44.416749 9578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: Bootstrap starting.
I20260620 19:16:44.417518 9578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:44.417807 9578 log.cc:824] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:44.425612 9578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: No bootstrap required, opened a new log
I20260620 19:16:44.426856 9695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8: Generated new TSK 0
I20260620 19:16:44.427261 9695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 19:16:44.427644 9578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.427790 9578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b767e7f61664112abf28c6b7fa92a80, State: Initialized, Role: FOLLOWER
I20260620 19:16:44.427874 9578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } }
I20260620 19:16:44.428187 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.428236 9578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:44.428319 9551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:44.429266 9551 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [term 1 FOLLOWER]: Refusing update from remote peer aa6401bb6c544fee86b3d6e8f0beb7f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:44.429816 9674 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 19:16:44.430094 9714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:44.430138 9714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:44.431712 9552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b767e7f61664112abf28c6b7fa92a80"
I20260620 19:16:44.429508 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.431947 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa6401bb6c544fee86b3d6e8f0beb7f8. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.432052 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.432119 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.432207 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:44.433926 9717 mvcc.cc:205] Tried to move back new op lower bound from 7299002386128330752 to 7299002385955594240. Current Snapshot: MvccSnapshot[applied={T|T < 7299002385997844480}]
I20260620 19:16:44.434127 9552 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate aa6401bb6c544fee86b3d6e8f0beb7f8 for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260620 19:16:44.456090 9505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.454784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.19.66.254:38317)
0620 19:16:44.454827 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.454829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.455433 (+ 604us) server_negotiation.cc:407] Connection header received
0620 19:16:44.455544 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.455545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.455557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.455586 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.456018 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55832: BlockingRecv error: recv got EOF from 127.0.0.1:55832 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 19:16:44.456964 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.455118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.19.66.253:40791)
0620 19:16:44.455156 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.455158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.456303 (+ 1145us) server_negotiation.cc:407] Connection header received
0620 19:16:44.456482 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.456483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.456493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.456521 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.456894 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57732: BlockingRecv error: recv got EOF from 127.0.0.1:57732 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:16:44.458094 9658 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.457286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36130 (local address 127.19.66.252:45391)
0620 19:16:44.457323 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.457325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.457440 (+ 115us) server_negotiation.cc:407] Connection header received
0620 19:16:44.457528 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.457529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.457538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.457565 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.458040 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36130: BlockingRecv error: recv got EOF from 127.0.0.1:36130 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:16:44.460315 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.459559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57744 (local address 127.19.66.253:40791)
0620 19:16:44.459591 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.459593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.459815 (+ 222us) server_negotiation.cc:407] Connection header received
0620 19:16:44.459916 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.459917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.459932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.459960 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.460273 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57744: BlockingRecv error: recv got EOF from 127.0.0.1:57744 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:16:44.463167 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.462410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57760 (local address 127.19.66.253:40791)
0620 19:16:44.462442 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.462444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.462693 (+ 249us) server_negotiation.cc:407] Connection header received
0620 19:16:44.462785 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.462786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.462796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.462823 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.463127 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57760: BlockingRecv error: recv got EOF from 127.0.0.1:57760 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:16:44.466936 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.466186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.19.66.253:40791)
0620 19:16:44.466220 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.466221 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.466454 (+ 233us) server_negotiation.cc:407] Connection header received
0620 19:16:44.466544 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.466545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.466555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.466583 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.466882 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57766: BlockingRecv error: recv got EOF from 127.0.0.1:57766 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:16:44.472735 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.471938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.19.66.253:40791)
0620 19:16:44.471972 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.471974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.472259 (+ 285us) server_negotiation.cc:407] Connection header received
0620 19:16:44.472353 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.472354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.472364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.472391 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.472691 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57772: BlockingRecv error: recv got EOF from 127.0.0.1:57772 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 19:16:44.482841 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.481958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.19.66.253:40791)
0620 19:16:44.481999 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.482001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.482286 (+ 285us) server_negotiation.cc:407] Connection header received
0620 19:16:44.482383 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.482384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.482396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.482424 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.482784 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57786: BlockingRecv error: recv got EOF from 127.0.0.1:57786 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 19:16:44.501072 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.500075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.19.66.253:40791)
0620 19:16:44.500120 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.500122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.500444 (+ 322us) server_negotiation.cc:407] Connection header received
0620 19:16:44.500551 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.500552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.500564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.500593 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.501010 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57790: BlockingRecv error: recv got EOF from 127.0.0.1:57790 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 19:16:44.534857 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.534173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57792 (local address 127.19.66.253:40791)
0620 19:16:44.534221 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.534223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.534356 (+ 133us) server_negotiation.cc:407] Connection header received
0620 19:16:44.534422 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.534423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.534435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.534480 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.534788 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57792: BlockingRecv error: recv got EOF from 127.0.0.1:57792 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:16:44.600922 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.600212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57804 (local address 127.19.66.253:40791)
0620 19:16:44.600258 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.600260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.600388 (+ 128us) server_negotiation.cc:407] Connection header received
0620 19:16:44.600455 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.600456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.600466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.600511 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.600835 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57804: BlockingRecv error: recv got EOF from 127.0.0.1:57804 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:16:44.730464 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.729788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.19.66.253:40791)
0620 19:16:44.729896 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.729899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.729955 (+ 56us) server_negotiation.cc:407] Connection header received
0620 19:16:44.730036 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.730038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.730049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.730096 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.730406 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57806: BlockingRecv error: recv got EOF from 127.0.0.1:57806 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 19:16:44.811326 9674 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa6401bb6c544fee86b3d6e8f0beb7f8" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 38317 } } peers { permanent_uuid: "6b767e7f61664112abf28c6b7fa92a80" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 40791 } } peers { permanent_uuid: "378e9d7e1ac54c47a64b6a759a89bc9c" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 45391 } } }
I20260620 19:16:44.811486 9674 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6401bb6c544fee86b3d6e8f0beb7f8 [sys.catalog]: This master's current role is: LEADER
W20260620 19:16:44.988181 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:44.987556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57810 (local address 127.19.66.253:40791)
0620 19:16:44.987630 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:44.987632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:44.987642 (+ 10us) server_negotiation.cc:407] Connection header received
0620 19:16:44.987718 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:44.987719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:44.987730 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:44.987777 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:44.988116 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57810: BlockingRecv error: recv got EOF from 127.0.0.1:57810 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 19:16:45.362505 9671 catalog_manager.cc:1259] Loaded cluster ID: 5770bdbfe294442db89b38568553a20a
I20260620 19:16:45.362579 9671 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: loading cluster ID for follower catalog manager: success
I20260620 19:16:45.363325 9671 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: acquiring CA information for follower catalog manager: success
I20260620 19:16:45.363582 9671 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 378e9d7e1ac54c47a64b6a759a89bc9c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:16:45.430518 9714 catalog_manager.cc:1259] Loaded cluster ID: 5770bdbfe294442db89b38568553a20a
I20260620 19:16:45.430591 9714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: loading cluster ID for follower catalog manager: success
I20260620 19:16:45.431358 9714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: acquiring CA information for follower catalog manager: success
I20260620 19:16:45.431622 9714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b767e7f61664112abf28c6b7fa92a80: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:16:45.503818 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:45.501361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.19.66.253:40791)
0620 19:16:45.501464 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:45.501465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:45.501476 (+ 11us) server_negotiation.cc:407] Connection header received
0620 19:16:45.501505 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:45.501505 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:45.501553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:45.501625 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:45.501811 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.502305 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.503070 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.503269 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.503320 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:45.503444 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:45.503555 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:45.503611 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:45.503700 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 19:16:45.503736 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260620 19:16:45.507946 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:45.505624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36144 (local address 127.19.66.252:45391)
0620 19:16:45.505720 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:45.505721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:45.505736 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:16:45.505788 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:45.505789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:45.505823 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:45.505889 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:45.505993 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.506466 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.507195 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.507408 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.507462 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:45.507589 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:45.507693 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:45.507777 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:45.507846 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 19:16:45.507862 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":49,"threads_started":1}
I20260620 19:16:45.512253 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:45.509664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.19.66.253:40791)
0620 19:16:45.509705 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:45.509706 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:45.509818 (+ 112us) server_negotiation.cc:407] Connection header received
0620 19:16:45.509904 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:45.509905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:45.509916 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:45.509941 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:45.510111 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.510818 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.511574 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.511823 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.511883 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:45.511986 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:45.512017 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:45.512071 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:45.512154 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 19:16:45.512175 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 19:16:45.512524 9793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:45.509564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.19.66.252:45391)
0620 19:16:45.509598 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:45.509600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:45.509664 (+ 64us) server_negotiation.cc:407] Connection header received
0620 19:16:45.509720 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:45.509720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:45.509731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:45.509770 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:45.509918 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.510621 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.511734 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.512024 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.512082 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:45.512214 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:45.512250 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:45.512291 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:45.512436 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 19:16:45.512456 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:45.513067 9800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:45.509542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.19.66.254:38317)
0620 19:16:45.509767 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:45.509769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:45.509835 (+ 66us) server_negotiation.cc:407] Connection header received
0620 19:16:45.509907 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:45.509908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:45.509958 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:45.510035 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:45.510143 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.510638 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.512125 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:45.512404 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:45.512462 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:45.512689 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:45.512815 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:45.512938 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:45.512985 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 19:16:45.513008 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":45,"threads_started":1}
I20260620 19:16:45.514608 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9435
I20260620 19:16:45.519490 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9510
I20260620 19:16:45.523584 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9585
2026-06-20T19:16:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2661 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 19:16:45 dist-test-slave-nq8r krb5kdc[9804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 19:16:45 dist-test-slave-nq8r krb5kdc[9804](info): set up 2 sockets
Jun 20 19:16:45 dist-test-slave-nq8r krb5kdc[9804](info): commencing operation
krb5kdc: starting...
W20260620 19:16:46.464767 19723 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/test-user.keytab.
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T19:16:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T19:16:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.19.66.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
Entry for principal kudu/127.19.66.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab.
I20260620 19:16:46.571342 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36371
--webserver_interface=127.19.66.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 19:16:46.650316 9817 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:46.650521 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:46.650537 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:46.650556 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:46.651633 9817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:46.651664 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:46.651676 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:46.651687 9817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:46.651703 9817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:46.653146 9817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.254.keytab
--principal=kudu/127.19.66.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.254:36371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.19.66.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:46.653337 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:46.653501 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:46.657828 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:46.657840 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:46.657840 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:46.658038 9817 server_base.cc:1034] running on GCE node
I20260620 19:16:46.658262 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:46.658511 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:46.659664 9817 hybrid_clock.cc:648] HybridClock initialized: now 1781983006659642 us; error 34 us; skew 500 ppm
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:46.664314 9817 init.cc:377] Logged in from keytab as kudu/127.19.66.254@KRBTEST.COM (short username kudu)
I20260620 19:16:46.664750 9817 webserver.cc:460] Webserver started at http://127.19.66.254:41735/ using document root <none> and password file <none>
I20260620 19:16:46.664978 9817 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:46.665026 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:46.665141 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:46.666196 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data/instance:
uuid: "b2c53f6831d34cab93d10d65d3b92e7c"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "b2616e76db984974dcd6e4ffbaac41a9"
server_key_iv: "0b4cf1a008578d9f149cf9c469f8e1f7"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.666522 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance:
uuid: "b2c53f6831d34cab93d10d65d3b92e7c"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "b2616e76db984974dcd6e4ffbaac41a9"
server_key_iv: "0b4cf1a008578d9f149cf9c469f8e1f7"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.667771 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 19:16:46.668434 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:46.668591 9817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 19:16:46.668674 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal
uuid: "b2c53f6831d34cab93d10d65d3b92e7c"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "b2616e76db984974dcd6e4ffbaac41a9"
server_key_iv: "0b4cf1a008578d9f149cf9c469f8e1f7"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.668762 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:46.686148 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:46.687968 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:46.688104 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:46.691840 9817 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.254:36371
I20260620 19:16:46.691888 9884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.254:36371 every 8 connection(s)
I20260620 19:16:46.692214 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/data/info.pb
I20260620 19:16:46.693183 9885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:46.694676 9885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.695784 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9817
I20260620 19:16:46.695879 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-0/wal/instance
I20260620 19:16:46.696075 19723 external_mini_cluster.cc:1428] Setting key 984b445cf1b2635ef6fcced590866b83
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.253@KRBTEST.COM; defaulting to no policy
I20260620 19:16:46.704773 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.695329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36371 (local address 127.0.0.1:50226)
0620 19:16:46.695489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.695524 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.696121 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.696403 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.696406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.696733 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.696983 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.696989 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.698009 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.698010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.698936 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.698938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.699435 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.699437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.699510 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.700012 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.700022 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.701598 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.703464 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.703470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.703472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.703650 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.703885 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.703886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.703886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.703967 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.704292 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.704295 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.704359 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.704425 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.704545 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":78,"threads_started":1}
I20260620 19:16:46.704778 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.695503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.19.66.254:36371)
0620 19:16:46.695619 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.695624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.695659 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:16:46.696281 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.696283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.696300 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.696376 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.697097 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.697909 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.699016 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.699335 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.699427 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.701684 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.701693 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.701693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.701701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.703377 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.703717 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.703721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.703721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.703824 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.704024 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.704025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.704025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.704242 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.704306 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.704461 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.704628 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":86,"thread_start_us":39,"threads_started":1}
I20260620 19:16:46.705474 9885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.705801 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.705697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51296)
0620 19:16:46.705738 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.705780 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:16:46.705912 9833 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.253@KRBTEST.COM" created.
I20260620 19:16:46.706272 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.706194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51300)
0620 19:16:46.706229 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.706250 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 19:16:46.706391 9885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:34151: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
Entry for principal kudu/127.19.66.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab.
I20260620 19:16:46.719419 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:34151
--webserver_interface=127.19.66.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:16:46.755618 9885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } attempt: 1
I20260620 19:16:46.756125 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.755982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51312)
0620 19:16:46.756047 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.756093 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 19:16:46.756445 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.756360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51320)
0620 19:16:46.756400 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.756426 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 19:16:46.756536 9885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:34151: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
W20260620 19:16:46.797950 9892 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:46.798154 9892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:46.798170 9892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:46.798187 9892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:46.799607 9892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:46.799650 9892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:46.799664 9892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:46.799675 9892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:46.799685 9892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:46.801155 9892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.253.keytab
--principal=kudu/127.19.66.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.253:34151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.19.66.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:46.801352 9892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:46.801523 9892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:46.805899 9898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:46.805896 9897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:46.805896 9900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:46.806105 9892 server_base.cc:1034] running on GCE node
I20260620 19:16:46.806313 9892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:46.806557 9892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:46.807690 9892 hybrid_clock.cc:648] HybridClock initialized: now 1781983006807679 us; error 29 us; skew 500 ppm
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:46.809978 9892 init.cc:377] Logged in from keytab as kudu/127.19.66.253@KRBTEST.COM (short username kudu)
I20260620 19:16:46.810384 9892 webserver.cc:460] Webserver started at http://127.19.66.253:33245/ using document root <none> and password file <none>
I20260620 19:16:46.810580 9892 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:46.810624 9892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:46.810746 9892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:46.811725 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data/instance:
uuid: "32724d2fb76e43fcae40ef803f2aa1ac"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "f62683676dfe8bed40bb8067f86e6ce9"
server_key_iv: "c8ed60a0f3458d65d4b98a7b45958900"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.812042 9892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal/instance:
uuid: "32724d2fb76e43fcae40ef803f2aa1ac"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "f62683676dfe8bed40bb8067f86e6ce9"
server_key_iv: "c8ed60a0f3458d65d4b98a7b45958900"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.813228 9892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 19:16:46.813858 9907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:46.814005 9892 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 19:16:46.814085 9892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal
uuid: "32724d2fb76e43fcae40ef803f2aa1ac"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "f62683676dfe8bed40bb8067f86e6ce9"
server_key_iv: "c8ed60a0f3458d65d4b98a7b45958900"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.814143 9892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:46.824663 9885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } attempt: 2
I20260620 19:16:46.825109 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.824985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51328)
0620 19:16:46.825040 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.825079 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 19:16:46.825392 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.825330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51336)
0620 19:16:46.825360 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.825379 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 19:16:46.825484 9885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.253:34151: Network error: Client connection negotiation failed: client connection to 127.19.66.253:34151: connect: Connection refused (error 111)
I20260620 19:16:46.835000 9892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:46.837061 9892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:46.837186 9892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:46.840864 9892 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.253:34151
I20260620 19:16:46.840919 9959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.253:34151 every 8 connection(s)
I20260620 19:16:46.841226 9892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/data/info.pb
I20260620 19:16:46.842098 9960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:46.843533 9960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.843901 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9892
I20260620 19:16:46.844007 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-1/wal/instance
I20260620 19:16:46.844177 19723 external_mini_cluster.cc:1428] Setting key dc0ca94d47d4a1c76a91aa4dd24446c3
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.19.66.252@KRBTEST.COM; defaulting to no policy
I20260620 19:16:46.852169 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.844232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.19.66.254:36371)
0620 19:16:46.844282 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.844284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.844519 (+ 235us) server_negotiation.cc:407] Connection header received
0620 19:16:46.845122 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.845123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.845133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.845185 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.845909 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.846388 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.847370 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.847699 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.847774 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.849738 (+ 1964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.849740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.849740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.849747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.851122 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.851464 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.851465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.851465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.851516 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.851686 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.851687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.851687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.851779 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.851825 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.852048 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.852107 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 19:16:46.852386 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.844208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36371 (local address 127.0.0.1:50234)
0620 19:16:46.844383 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.844433 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.845030 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.845257 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.845262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.845588 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.845816 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.845821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.846465 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.846466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.847249 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.847252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.847786 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.847787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.847887 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.848351 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.848358 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.849646 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.851200 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.851206 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.851207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.851405 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.851563 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.851563 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.851564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.851645 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.851821 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.851825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.851934 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.852027 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.852169 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
I20260620 19:16:46.852926 9960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.19.66.252@KRBTEST.COM" created.
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:46.859549 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.853162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51348)
0620 19:16:46.853201 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.853210 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.853242 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.853478 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.853478 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.853597 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.853640 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.853641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.854432 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.854433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.854982 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.854983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.855550 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.855551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.855620 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.855930 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.855935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.857096 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.858529 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.858530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.858531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.858736 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.858919 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.858920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.858920 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.858993 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.859208 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.859208 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.859251 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.859323 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.859412 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 19:16:46.859566 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.853218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.19.66.253:34151)
0620 19:16:46.853359 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.853363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.853385 (+ 22us) server_negotiation.cc:407] Connection header received
0620 19:16:46.853401 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.853402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.853412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.853470 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.853698 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.854354 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.855049 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.855476 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.855539 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.857171 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.857178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.857178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.857183 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.858454 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.858806 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.858808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.858808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.858867 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.859045 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.859045 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.859045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.859154 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.859206 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.859348 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.859462 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260620 19:16:46.860122 9960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.860409 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.860299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39802)
0620 19:16:46.860340 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.860386 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:16:46.860529 9911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
I20260620 19:16:46.860842 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.860757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39812)
0620 19:16:46.860800 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.860823 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:16:46.860956 9960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:34269: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
Entry for principal kudu/127.19.66.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab.
I20260620 19:16:46.865712 19723 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskiAqWb6/build/release/bin/kudu
/tmp/dist-test-taskiAqWb6/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:34269
--webserver_interface=127.19.66.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.19.66.212:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 19:16:46.910172 9960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } attempt: 1
I20260620 19:16:46.910668 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.910528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39816)
0620 19:16:46.910589 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.910636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 19:16:46.911000 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.910917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39828)
0620 19:16:46.910955 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.910981 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 19:16:46.911090 9960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:34269: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
I20260620 19:16:46.916625 9885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } attempt: 3
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:46.923372 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.916932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51364)
0620 19:16:46.916993 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.917006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.917068 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.917262 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.917263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.917428 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.917494 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.917496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.918405 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.918406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.918959 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.918961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.919436 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.919436 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.919485 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.919886 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.919891 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.921037 (+ 1146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.922549 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.922550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.922550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.922699 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.922895 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.922896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.922896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.922939 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.923149 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.923150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.923185 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.923225 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.923291 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 19:16:46.923413 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.917013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51364 (local address 127.19.66.253:34151)
0620 19:16:46.917075 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.917078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.917090 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:16:46.917119 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.917120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.917132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.917188 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.917563 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.918320 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.919031 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.919369 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.919434 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.921110 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.921112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.921112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.921120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.922481 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.922760 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.922761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.922762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.922823 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.922976 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.922976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.922977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.923098 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.923152 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.923248 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.923323 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:16:46.923792 9885 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.924104 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.924016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39834)
0620 19:16:46.924056 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.924082 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 19:16:46.924399 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.924346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39848)
0620 19:16:46.924367 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.924386 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 19:16:46.924487 9885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:34269: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
W20260620 19:16:46.945333 9967 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 19:16:46.945528 9967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 19:16:46.945544 9967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 19:16:46.945561 9967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 19:16:46.946628 9967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 19:16:46.946660 9967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 19:16:46.946671 9967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 19:16:46.946682 9967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 19:16:46.946695 9967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 19:16:46.948067 9967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.19.66.212:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.19.66.254:36371,127.19.66.253:34151,127.19.66.252:34269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/krb5kdc/kudu_127.19.66.252.keytab
--principal=kudu/127.19.66.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.19.66.252:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.19.66.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 17:47:57 UTC on ccycloud.quasar-jlmybm.root.comops.site
build id 4457640
I20260620 19:16:46.948251 9967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 19:16:46.948427 9967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 19:16:46.952806 9972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 19:16:46.952841 9973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:46.953016 9967 server_base.cc:1034] running on GCE node
W20260620 19:16:46.952960 9975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 19:16:46.953298 9967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 19:16:46.953547 9967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 19:16:46.954686 9967 hybrid_clock.cc:648] HybridClock initialized: now 1781983006954670 us; error 30 us; skew 500 ppm
I20260620 19:16:46.955612 9885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } attempt: 1
I20260620 19:16:46.955938 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.955829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39852)
0620 19:16:46.955882 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.955910 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 19:16:46.956265 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.956148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39858)
0620 19:16:46.956199 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.956233 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 19:16:46.956380 9885 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.19.66.252:34269: Network error: Client connection negotiation failed: client connection to 127.19.66.252:34269: connect: Connection refused (error 111)
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 19:16:46.957103 9967 init.cc:377] Logged in from keytab as kudu/127.19.66.252@KRBTEST.COM (short username kudu)
I20260620 19:16:46.957490 9967 webserver.cc:460] Webserver started at http://127.19.66.252:35763/ using document root <none> and password file <none>
I20260620 19:16:46.957680 9967 fs_manager.cc:359] Metadata directory not provided
I20260620 19:16:46.957722 9967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 19:16:46.957834 9967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 19:16:46.958827 9967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data/instance:
uuid: "70b777f7763d415aaf6785ff31d8ee13"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "fcbc0793fb8ad3be224c029f317ca5a5"
server_key_iv: "5925b20abb74718ffb7398e786f77a1e"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.959142 9967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal/instance:
uuid: "70b777f7763d415aaf6785ff31d8ee13"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "fcbc0793fb8ad3be224c029f317ca5a5"
server_key_iv: "5925b20abb74718ffb7398e786f77a1e"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.960341 9967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 19:16:46.960956 9982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:46.961086 9967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 19:16:46.961166 9967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data,/tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal
uuid: "70b777f7763d415aaf6785ff31d8ee13"
format_stamp: "Formatted at 2026-06-20 19:16:46 on dist-test-slave-nq8r"
server_key: "fcbc0793fb8ad3be224c029f317ca5a5"
server_key_iv: "5925b20abb74718ffb7398e786f77a1e"
server_key_version: "encryptionkey@0"
I20260620 19:16:46.961225 9967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 19:16:46.969596 9967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 19:16:46.971213 9967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 19:16:46.971333 9967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 19:16:46.975564 9967 rpc_server.cc:307] RPC server started. Bound to: 127.19.66.252:34269
I20260620 19:16:46.975595 10034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.19.66.252:34269 every 8 connection(s)
I20260620 19:16:46.976042 9967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/data/info.pb
I20260620 19:16:46.977183 10035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 19:16:46.979049 10035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.979231 9960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } attempt: 2
I20260620 19:16:46.979699 19723 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskiAqWb6/build/release/bin/kudu as pid 9967
I20260620 19:16:46.979809 19723 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskiAqWb6/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781982751812665-19723-0/minicluster-data/master-2/wal/instance
I20260620 19:16:46.980001 19723 external_mini_cluster.cc:1428] Setting key d6962db9d1a0f994086628b51b568f8f
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.253@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:46.988641 9962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.979523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39860)
0620 19:16:46.979594 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.979605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.979673 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.980807 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.980808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.980995 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.981040 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.981042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.982526 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.982527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.983272 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.983275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.983768 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.983824 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.984252 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.984257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.985547 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.987292 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.987293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.987293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.987442 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.987680 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.987681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.987681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.987750 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.988099 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.988330 (+ 231us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.988378 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.988454 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.988543 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 19:16:46.989064 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.980177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39860 (local address 127.19.66.252:34269)
0620 19:16:46.980526 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.980530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.980565 (+ 35us) server_negotiation.cc:407] Connection header received
0620 19:16:46.980677 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.980685 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.980710 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.980824 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.981134 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.982428 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983360 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.983685 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983989 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.985624 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.985631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.985632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.985639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.987228 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.987505 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.987508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.987508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.987603 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.987812 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.987813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.987813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.988027 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.988101 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.988538 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.988655 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":320,"thread_start_us":65,"threads_started":1}
I20260620 19:16:46.989308 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.979711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.254:36371 (local address 127.0.0.1:50242)
0620 19:16:46.979883 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.980039 (+ 156us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.980677 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.980964 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.980966 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.981256 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.981359 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.981363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.982232 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.982233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.983309 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.983311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.983760 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.983869 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.984395 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.984404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.986571 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.988201 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.988205 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.988207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.988472 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.988660 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.988661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.988661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.988714 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.989000 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.989006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.989044 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.989106 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.989205 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260620 19:16:46.989574 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.979967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50242 (local address 127.19.66.254:36371)
0620 19:16:46.980006 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.980008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.980126 (+ 118us) server_negotiation.cc:407] Connection header received
0620 19:16:46.980798 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.980799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.980811 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.980840 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.981456 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.982163 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983393 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.983670 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983732 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.986647 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.986648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.986649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.986656 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.988141 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.988543 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.988545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.988545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.988601 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.988780 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.988781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.988781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.988921 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.988976 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.989423 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.989493 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:46.989976 10035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } has no permanent_uuid. Determining permanent_uuid...
I20260620 19:16:46.990679 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.980326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.19.66.252:34269)
0620 19:16:46.980738 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.980739 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.981613 (+ 874us) server_negotiation.cc:407] Connection header received
0620 19:16:46.981631 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.981632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.981642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.981694 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.982357 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.982835 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983553 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.983844 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.983897 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.987059 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.987060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.987060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.987067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.989046 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.989623 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.989624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.989624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.989683 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.989946 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.989947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.989948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.990063 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.990185 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.990496 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.990559 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":391,"thread_start_us":70,"threads_started":1}
I20260620 19:16:46.991055 9960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: Bootstrap starting.
I20260620 19:16:46.992005 9960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:46.992293 9960 log.cc:824] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:46.993325 9960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: No bootstrap required, opened a new log
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:46.995290 9960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:46.995463 9960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32724d2fb76e43fcae40ef803f2aa1ac, State: Initialized, Role: FOLLOWER
I20260620 19:16:46.995587 9960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.254@KRBTEST.COM
I20260620 19:16:46.995947 9960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:46.996166 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.990242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.19.66.253:34151)
0620 19:16:46.990279 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.990281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.990287 (+ 6us) server_negotiation.cc:407] Connection header received
0620 19:16:46.990327 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.990327 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.990336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.990382 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.990637 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.991287 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.992092 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.992283 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.992332 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.993942 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.993943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.993944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.993950 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.995144 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.995419 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.995420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.995421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.995470 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.995643 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.995643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.995644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.995788 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.995851 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.996023 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 19:16:46.996085 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:16:46.996166 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.990207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.253:34151 (local address 127.0.0.1:51366)
0620 19:16:46.990246 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.990255 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.990283 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.990394 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.990395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.990523 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.990555 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.990556 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.991383 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.991384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.992026 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.992028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.992342 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.992343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.992398 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.992744 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:46.992748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:46.993872 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:46.995216 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.995216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.995217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.995363 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.995535 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:46.995535 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:46.995536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:46.995588 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:46.995876 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:46.995876 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:46.995914 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 19:16:46.995993 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 19:16:46.996075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 19:16:46.996491 10035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } has no permanent_uuid. Determining permanent_uuid...
W20260620 19:16:46.997815 10053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:46.998497 10053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:46.999735 10042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.000113 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.991575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.19.66.254:36371)
0620 19:16:46.991609 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.991610 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.991617 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:46.991635 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.991635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.991643 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.991695 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.992213 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.992781 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.993663 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.993922 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.993959 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:46.996582 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:46.996583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:46.996584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:46.996591 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:46.998168 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.999404 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.999405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.999405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.999473 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:46.999651 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:46.999652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:46.999652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:46.999776 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:46.999925 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:46.999960 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.000030 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 19:16:47.000412 10042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 19:16:46 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.252@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:46.999001 9892 master_runner.cc:186] Error getting master registration for 127.19.66.254:36371: OK, instance_id { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" instance_seqno: 1781983006690816 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:16:47.001854 9892 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:16:47.003932 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.996998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39880)
0620 19:16:46.997342 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:46.997351 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:46.997404 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:46.997533 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:46.997534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:46.997662 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:46.997691 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.997692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.998499 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.998500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.999004 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:46.999005 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.999448 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.999449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:46.999493 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.000212 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:47.000216 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:47.001309 (+ 1093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:47.002933 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:47.002934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:47.002934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:47.003116 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:47.003305 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:47.003306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:47.003306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:47.003372 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:47.003668 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:47.003668 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:47.003699 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 19:16:47.003746 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 19:16:47.003836 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329}
I20260620 19:16:47.003990 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:46.996962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.19.66.252:34269)
0620 19:16:46.997003 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:46.997005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:46.997395 (+ 390us) server_negotiation.cc:407] Connection header received
0620 19:16:46.997457 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:46.997458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:46.997465 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:46.997489 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:46.997761 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.998425 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.999074 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:46.999355 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:46.999410 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.001377 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.001378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.001378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.001384 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.002852 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.003182 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.003183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.003184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.003240 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.003427 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.003427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.003427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.003592 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.003647 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.003794 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.003892 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":21}
I20260620 19:16:47.005446 10035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: Bootstrap starting.
I20260620 19:16:47.006021 10035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:47.006271 10035 log.cc:824] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: Log is configured to *not* fsync() on all Append() calls
Jun 20 19:16:47 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.19.66.253@KRBTEST.COM
I20260620 19:16:47.006928 10035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: No bootstrap required, opened a new log
I20260620 19:16:47.008279 10035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.008392 10035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b777f7763d415aaf6785ff31d8ee13, State: Initialized, Role: FOLLOWER
I20260620 19:16:47.008445 10035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.008684 10035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:47.008917 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.008999 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.011394 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.002364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.19.66.253:34151)
0620 19:16:47.002396 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.002398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.002405 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:47.002443 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.002444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.002452 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.002502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.002977 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.003551 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.004485 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.004774 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.004833 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.008395 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.008396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.008396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.008403 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.009951 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.010318 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.010319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.010319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.010386 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.010568 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.010569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.010569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.010687 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.010810 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.011213 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.011284 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 19:16:47.010658 9967 master_runner.cc:186] Error getting master registration for 127.19.66.254:36371: OK, instance_id { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" instance_seqno: 1781983006690816 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 19:16:47.012809 9967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 19:16:47.013237 10067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 19:16:47.013329 10067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 19:16:47.014158 19723 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 19:16:47.015239 9859 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b2c53f6831d34cab93d10d65d3b92e7c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50256
I20260620 19:16:47.031546 9885 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } attempt: 2
Jun 20 19:16:47 dist-test-slave-nq8r krb5kdc[9804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781983006, etypes {rep=17 tkt=17 ses=17}, kudu/127.19.66.254@KRBTEST.COM for kudu/127.19.66.252@KRBTEST.COM
I20260620 19:16:47.039525 9887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.031909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.19.66.252:34269 (local address 127.0.0.1:39886)
0620 19:16:47.032040 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 19:16:47.032051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 19:16:47.032094 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 19:16:47.032249 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 19:16:47.032251 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 19:16:47.032446 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 19:16:47.032503 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:47.032506 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.033446 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.033448 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:47.034252 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 19:16:47.034255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.034687 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.034688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 19:16:47.034752 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.035226 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 19:16:47.035230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 19:16:47.036680 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 19:16:47.038468 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:47.038469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:47.038470 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:47.038696 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:47.038905 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 19:16:47.038905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 19:16:47.038906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 19:16:47.038968 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 19:16:47.039196 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 19:16:47.039197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 19:16:47.039253 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 19:16:47.039318 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 19:16:47.039418 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260620 19:16:47.039525 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.032065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.19.66.252:34269)
0620 19:16:47.032112 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.032114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.032126 (+ 12us) server_negotiation.cc:407] Connection header received
0620 19:16:47.032154 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.032154 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.032164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.032229 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.032574 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.033364 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.034340 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.034623 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.034690 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.036763 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.036765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.036765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.036772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.038395 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.038767 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.038768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.038768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.038826 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.039023 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.039024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.039024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.039148 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.039208 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.039345 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.039430 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 19:16:47.041620 9885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c: Bootstrap starting.
I20260620 19:16:47.042357 9885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c: Neither blocks nor log segments found. Creating new log.
I20260620 19:16:47.042649 9885 log.cc:824] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c: Log is configured to *not* fsync() on all Append() calls
I20260620 19:16:47.043506 9885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c: No bootstrap required, opened a new log
I20260620 19:16:47.044888 9885 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.045038 9885 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c53f6831d34cab93d10d65d3b92e7c, State: Initialized, Role: FOLLOWER
I20260620 19:16:47.045120 9885 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.045464 9859 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 19:16:47.045574 9859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:47.046365 9859 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.046617 9859 leader_election.cc:288] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [CANDIDATE]: Term 1 election: Requested vote from peers 32724d2fb76e43fcae40ef803f2aa1ac (127.19.66.253:34151), 70b777f7763d415aaf6785ff31d8ee13 (127.19.66.252:34269)
I20260620 19:16:47.045454 9885 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 19:16:47.047727 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "70b777f7763d415aaf6785ff31d8ee13"
I20260620 19:16:47.047854 10001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:47.045471 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.048000 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.048699 10001 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2c53f6831d34cab93d10d65d3b92e7c in term 1.
I20260620 19:16:47.049098 9934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "32724d2fb76e43fcae40ef803f2aa1ac"
I20260620 19:16:47.049227 9934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [term 0 FOLLOWER]: Advancing to term 1
I20260620 19:16:47.050029 9934 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2c53f6831d34cab93d10d65d3b92e7c in term 1.
I20260620 19:16:47.050274 9836 leader_election.cc:302] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b777f7763d415aaf6785ff31d8ee13, b2c53f6831d34cab93d10d65d3b92e7c; no voters:
I20260620 19:16:47.050398 10070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 19:16:47.050549 10070 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [term 1 LEADER]: Becoming Leader. State: Replica: b2c53f6831d34cab93d10d65d3b92e7c, State: Running, Role: LEADER
I20260620 19:16:47.050630 10070 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } }
I20260620 19:16:47.051036 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c53f6831d34cab93d10d65d3b92e7c. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.051102 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:47.051251 10088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 19:16:47.051899 10088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 19:16:47.052182 9817 master_runner.cc:263] No leader master found from master b2c53f6831d34cab93d10d65d3b92e7c
I20260620 19:16:47.052223 9817 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 19:16:47.063542 9934 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [term 1 FOLLOWER]: Refusing update from remote peer b2c53f6831d34cab93d10d65d3b92e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:47.063618 10001 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [term 1 FOLLOWER]: Refusing update from remote peer b2c53f6831d34cab93d10d65d3b92e7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 19:16:47.063838 10070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:47.064131 10070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 19:16:47.065928 10042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c53f6831d34cab93d10d65d3b92e7c. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.066018 10042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.066774 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2c53f6831d34cab93d10d65d3b92e7c. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.066859 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.069015 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.052500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.19.66.252:34269)
0620 19:16:47.052547 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.052550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.053193 (+ 643us) server_negotiation.cc:407] Connection header received
0620 19:16:47.053285 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.053286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.053296 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.053322 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.054068 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.054809 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.055843 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.056125 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.056165 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.062523 (+ 6358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.062525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.062526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.062533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.064836 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.067312 (+ 2476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.067313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.067313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.067384 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.068022 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.068023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.068023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.068153 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.068242 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.068822 (+ 580us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.068921 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 19:16:47.070967 9888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.052902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.19.66.254:36371)
0620 19:16:47.052944 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.052946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.056725 (+ 3779us) server_negotiation.cc:407] Connection header received
0620 19:16:47.056815 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.056816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.056825 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.056875 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.057567 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.058271 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.059295 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.059597 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.059633 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.065532 (+ 5899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.065533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.065534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.065541 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.067167 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.068554 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.068555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.068555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.068612 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.070360 (+ 1748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.070361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.070361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.070503 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.070572 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.070789 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.070874 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 19:16:47.071619 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:47.053085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.19.66.253:34151)
0620 19:16:47.053121 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.053123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.060174 (+ 7051us) server_negotiation.cc:407] Connection header received
0620 19:16:47.060229 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.060229 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.060240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.060299 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.061025 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.061682 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.066993 (+ 5311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:47.067308 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:47.067347 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:47.068787 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 19:16:47.068789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 19:16:47.068789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 19:16:47.068796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 19:16:47.070323 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.070685 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.070686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.070686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.070738 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 19:16:47.070928 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 19:16:47.070929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 19:16:47.070929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 19:16:47.071065 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 19:16:47.071154 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:47.071446 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 19:16:47.071529 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 19:16:47.073210 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.073302 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:47.073521 10070 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.073578 10070 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c [sys.catalog]: This master's current role is: LEADER
I20260620 19:16:47.073535 10042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.073611 10042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.073810 10090 mvcc.cc:205] Tried to move back new op lower bound from 7299002396929478656 to 7299002396880105472. Current Snapshot: MvccSnapshot[applied={T|T < 7299002396929478656}]
I20260620 19:16:47.073916 10094 mvcc.cc:205] Tried to move back new op lower bound from 7299002396929478656 to 7299002396880105472. Current Snapshot: MvccSnapshot[applied={T|T < 7299002396929478656}]
I20260620 19:16:47.074066 10056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2c53f6831d34cab93d10d65d3b92e7c" member_type: VOTER last_known_addr { host: "127.19.66.254" port: 36371 } } peers { permanent_uuid: "32724d2fb76e43fcae40ef803f2aa1ac" member_type: VOTER last_known_addr { host: "127.19.66.253" port: 34151 } } peers { permanent_uuid: "70b777f7763d415aaf6785ff31d8ee13" member_type: VOTER last_known_addr { host: "127.19.66.252" port: 34269 } } }
I20260620 19:16:47.074132 10056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 19:16:47.074043 10096 mvcc.cc:205] Tried to move back new op lower bound from 7299002396929478656 to 7299002396880105472. Current Snapshot: MvccSnapshot[applied={T|T < 7299002396929478656}]
I20260620 19:16:47.081363 10088 catalog_manager.cc:1347] Generated new cluster ID: 8b6943edcd384b599d0afdc6ec44ce44
I20260620 19:16:47.081441 10088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 19:16:47.100946 10088 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 19:16:47.101678 10088 catalog_manager.cc:1504] Loading token signing keys...
I20260620 19:16:47.109513 10088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2c53f6831d34cab93d10d65d3b92e7c: Generated new TSK 0
I20260620 19:16:47.109740 10088 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 19:16:47.136127 10038 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.135092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.19.66.252:34269)
0620 19:16:47.135137 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.135139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.135458 (+ 319us) server_negotiation.cc:407] Connection header received
0620 19:16:47.135558 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.135559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.135570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.135599 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.136062 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39898: BlockingRecv error: recv got EOF from 127.0.0.1:39898 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 19:16:47.137341 9888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.136513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.19.66.254:36371)
0620 19:16:47.136564 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.136566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.136774 (+ 208us) server_negotiation.cc:407] Connection header received
0620 19:16:47.136911 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.136912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.136925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.136955 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.137274 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50272: BlockingRecv error: recv got EOF from 127.0.0.1:50272 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 19:16:47.138084 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.136653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.19.66.253:34151)
0620 19:16:47.136694 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.136695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.137579 (+ 884us) server_negotiation.cc:407] Connection header received
0620 19:16:47.137670 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.137672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.137683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.137712 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.138026 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51402: BlockingRecv error: recv got EOF from 127.0.0.1:51402 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 19:16:47.140200 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.139524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.19.66.253:34151)
0620 19:16:47.139557 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.139559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.139723 (+ 164us) server_negotiation.cc:407] Connection header received
0620 19:16:47.139815 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.139816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.139832 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.139857 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.140156 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51408: BlockingRecv error: recv got EOF from 127.0.0.1:51408 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:16:47.143136 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.142421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.19.66.253:34151)
0620 19:16:47.142453 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.142454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.142652 (+ 198us) server_negotiation.cc:407] Connection header received
0620 19:16:47.142741 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.142742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.142751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.142776 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.143094 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51410: BlockingRecv error: recv got EOF from 127.0.0.1:51410 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 19:16:47.146962 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.146242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51420 (local address 127.19.66.253:34151)
0620 19:16:47.146275 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.146277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.146483 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:16:47.146574 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.146574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.146584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.146610 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.146917 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51420: BlockingRecv error: recv got EOF from 127.0.0.1:51420 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 19:16:47.152945 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.152140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.19.66.253:34151)
0620 19:16:47.152184 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.152186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.152392 (+ 206us) server_negotiation.cc:407] Connection header received
0620 19:16:47.152481 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.152483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.152492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.152518 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.152832 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51430: BlockingRecv error: recv got EOF from 127.0.0.1:51430 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 19:16:47.163040 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.162192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.19.66.253:34151)
0620 19:16:47.162239 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.162241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.162489 (+ 248us) server_negotiation.cc:407] Connection header received
0620 19:16:47.162587 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.162589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.162600 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.162628 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.162988 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51434: BlockingRecv error: recv got EOF from 127.0.0.1:51434 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 19:16:47.180881 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.180214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51444 (local address 127.19.66.253:34151)
0620 19:16:47.180263 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.180265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.180322 (+ 57us) server_negotiation.cc:407] Connection header received
0620 19:16:47.180407 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.180408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.180421 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.180479 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.180784 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51444: BlockingRecv error: recv got EOF from 127.0.0.1:51444 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 19:16:47.214498 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.213870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.19.66.253:34151)
0620 19:16:47.213937 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.213939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.214001 (+ 62us) server_negotiation.cc:407] Connection header received
0620 19:16:47.214054 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.214055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.214066 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.214112 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.214439 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51446: BlockingRecv error: recv got EOF from 127.0.0.1:51446 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 19:16:47.280290 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.279650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.19.66.253:34151)
0620 19:16:47.279730 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.279732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.279792 (+ 60us) server_negotiation.cc:407] Connection header received
0620 19:16:47.279830 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.279831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.279855 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.279933 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.280231 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51460: BlockingRecv error: recv got EOF from 127.0.0.1:51460 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 19:16:47.410071 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.409328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.19.66.253:34151)
0620 19:16:47.409405 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.409407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.409508 (+ 101us) server_negotiation.cc:407] Connection header received
0620 19:16:47.409571 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.409572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.409584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.409638 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.410007 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51474: BlockingRecv error: recv got EOF from 127.0.0.1:51474 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 19:16:47.667599 9964 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 19:16:47.667005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.19.66.253:34151)
0620 19:16:47.667061 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:47.667063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:47.667086 (+ 23us) server_negotiation.cc:407] Connection header received
0620 19:16:47.667153 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:47.667154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:47.667167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:47.667226 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 19:16:47.667540 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51484: BlockingRecv error: recv got EOF from 127.0.0.1:51484 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 19:16:47.998976 10053 catalog_manager.cc:1259] Loaded cluster ID: 8b6943edcd384b599d0afdc6ec44ce44
I20260620 19:16:47.999047 10053 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: loading cluster ID for follower catalog manager: success
I20260620 19:16:47.999830 10053 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: acquiring CA information for follower catalog manager: success
I20260620 19:16:48.000128 10053 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 32724d2fb76e43fcae40ef803f2aa1ac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:16:48.013734 10067 catalog_manager.cc:1259] Loaded cluster ID: 8b6943edcd384b599d0afdc6ec44ce44
I20260620 19:16:48.013799 10067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: loading cluster ID for follower catalog manager: success
I20260620 19:16:48.014586 10067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: acquiring CA information for follower catalog manager: success
I20260620 19:16:48.014863 10067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 70b777f7763d415aaf6785ff31d8ee13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 19:16:48.182999 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:48.180593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.19.66.253:34151)
0620 19:16:48.180709 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:48.180711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:48.180737 (+ 26us) server_negotiation.cc:407] Connection header received
0620 19:16:48.180803 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:48.180804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:48.180870 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:48.180946 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:48.181094 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.181598 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.182334 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.182542 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.182601 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:48.182706 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:48.182812 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:48.182868 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:48.182900 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 19:16:48.182917 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260620 19:16:48.186722 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:48.184684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.19.66.252:34269)
0620 19:16:48.184773 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:48.184775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:48.184782 (+ 7us) server_negotiation.cc:407] Connection header received
0620 19:16:48.184817 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:48.184818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:48.184870 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:48.184942 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:48.185018 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.185473 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.186104 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.186288 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.186334 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:48.186450 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:48.186545 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:48.186596 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:48.186614 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 19:16:48.186629 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":44,"threads_started":1}
I20260620 19:16:48.190603 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:48.188273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.19.66.252:34269)
0620 19:16:48.188306 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:48.188308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:48.188316 (+ 8us) server_negotiation.cc:407] Connection header received
0620 19:16:48.188357 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:48.188358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:48.188367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:48.188424 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:48.188493 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.189061 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.189875 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.190152 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.190206 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:48.190368 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:48.190399 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:48.190447 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:48.190520 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 19:16:48.190541 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 19:16:48.191028 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:48.188195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.19.66.254:36371)
0620 19:16:48.188333 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:48.188335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:48.188350 (+ 15us) server_negotiation.cc:407] Connection header received
0620 19:16:48.188390 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:48.188390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:48.188438 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:48.188509 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:48.188618 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.189422 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.190196 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.190443 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.190497 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:48.190644 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:48.190740 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:48.190788 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:48.190867 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 19:16:48.190951 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":42,"threads_started":1}
I20260620 19:16:48.191423 10164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 19:16:48.188191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.19.66.253:34151)
0620 19:16:48.188251 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 19:16:48.188253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 19:16:48.188626 (+ 373us) server_negotiation.cc:407] Connection header received
0620 19:16:48.188678 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 19:16:48.188678 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 19:16:48.188689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 19:16:48.188742 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 19:16:48.188839 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.189550 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.190650 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 19:16:48.190928 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 19:16:48.190984 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 19:16:48.191148 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 19:16:48.191183 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 19:16:48.191230 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 19:16:48.191350 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 19:16:48.191371 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 19:16:48.192553 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9817
I20260620 19:16:48.197526 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9892
I20260620 19:16:48.201808 19723 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskiAqWb6/build/release/bin/kudu with pid 9967
2026-06-20T19:16:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2678 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5339 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (256391 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST